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Introduction 


The  Environmental  Security  Technology  Certification  Program  (ESTCP) 
funded  the  Naval  Explosive  Ordnance  Disposal  Technology  Division 
(NAVEODTECHDIV)  (lead  agency),  the  U.S.  Army  Engineer  Research  and 
Development  Center  (ERDC),  and  the  U.S.  Army  Environmental  Center  (AEC) 
to  design  and  conduct  controlled  technology  demonstrations  at  the  U.S.  Army 
Jefferson  Proving  Ground  (JPG)  in  Madison,  Indiana.  These  technology 
demonstrations  were  conducted  during  the  period  June  to  November  2000  at 
three  1 -hectare  areas  located  near  the  test  site  used  during  the  JPG  Phase  IV 
demonstrations  (NAVEODTECHDIV  2000a)).  The  demonstrations  were 
designed  to  assess  the  capabilities  of  state-of-the-art  technologies  to  detect, 
discriminate,  and  identify  unexploded  ordnance  (UXO)  in  areas  containing 
natural  (magnetic  rocks/soils)  and  man-made  (munitions  fragments)  clutter.  This 
report  documents  the  results  of  these  demonstrations  and  is  intended  to  aid  the 
Government  in  selecting  effective  and  efficient  systems  for  UXO  detection  and 
discrimination  in  difficult  magnetic  clutter  sites  such  as  those  encountered  at 
Kaho'olawe,  Hawaii. 


Background  Information 

The  Department  of  Defense  (DOD)  is  currently  involved  in  a  number  of 
UXO  site  remediation  efforts  where  rapid  transition  of  advanced  technologies 
can  save  substantial  sums  of  money,  improve  UXO  detection  efficiency,  and 
significantly  expedite  the  transfer  of  lands  for  reuse.  One  of  the  most  prominent 
of  these  efforts  is  the  ongoing  UXO  cleanup  of  the  Kaho'olawe  bombing  ranges. 
The  major  difficulty  with  this  site  is  that  the  significant  magnetic  anomalies  from 
geologic  sources  and  near-surface  fragments  make  traditional  magnetometer- 
based  surveys  impractical.  Even  surveys  conducted  with  commercially  available 
electromagnetic  induction  (EMI)  instruments  such  as  the  Geonics  EM-61  have 
performed  ineffectively  in  these  conditions.  As  of  1  March  2000,  contractors  at 
Kaho’olawe  have  detected  12,121  subsurface  anomalies,  and  after  digging,  they 
have  found  that  only  4  percent  are  UXO,  32  percent  are  false  positives  resulting 
from  geologic  variations,  and  64  percent  are  the  result  of  buried  metal  from  both 
UXO  and  non-UXO-related  materials  (NAVEODTECHDIV  2000b).  The  focus 
of  this  project  is  to  evaluate  the  top  sensing  technologies  identified  during  pre¬ 
vious  JPG  demonstrations,  but  under  more  realistic  conditions,  in  order  to  quan¬ 
tify  their  detection,  discrimination,  cost,  and  production  rates  while  operating  at 
several  areas  within  JPG  that  contain  varying  degrees  of  geologic  magnetic  noise. 
The  purpose  of  this  report  is  to  aid  managers  of  UXO  cleanup  projects,  as  well  as 
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regulators  and  other  stakeholders,  to  make  informed  decisions  concerning  the 
capabilities,  costs,  and  risks  associated  with  applying  these  technologies  to  their 
site-specific  UXO  remediation  projects.  A  list  of  points  of  contact  from  the 
regulatory  and  other  communities  who  are  aware  of  this  demonstration  project  is 
provided  in  Appendix  A.  Following  in-depth  evaluation  of  performance  at  the 
JPG  site,  ESTCP  plans  to  transition  the  most  promising  technologies  to 
Kaho’olawe  for  additional  demonstrations  at  controlled  and  live  sites  during 
FY01. 

This  project  was  designed  to  incorporate  the  lessons  learned  from  previous 
UXO  technology  demonstrations  and  to  extend  the  results  of  the  JPG  Phase  IV 
Demonstrations  that  were  completed  during  FY  97.  The  JPG  Phase  IV  results 
indicated  that  advanced  UXO  sensing  and  processing  technologies  have  the 
potential  to  significantly  reduce  the  number  of  false  alarms.  Unfortunately,  those 
demonstrations  incorporated  a  number  of  artificial  factors  that  limited  the  validity 
of  the  conclusions  that  could  be  determined  from  the  results.  Some  of  the  arti¬ 
ficialities  included  the  use  of  nonrealistic  clutter  items,  the  fact  that  all  of  the 
clutter  items  were  made  available  to  the  demonstrators  for  system  training  prior 
to  the  field  tests,  and  the  lack  of  wide  area  search  requirements  (i.e.,  target  loca¬ 
tions  were  provided  to  the  demonstrators).  In  addition,  JPG  Phase  IV  demon¬ 
strations  did  not  provide  the  operational  performance  data  required  to  quantify 
the  cost  savings  and  risks  associated  with  using  these  technologies  in  actual 
cleanup  operations. 


Official  DOD  Requirement  Statement(s) 

This  project  addresses  the  Tri-Service  Environmental  Quality  Research, 
Development,  Test,  and  Evaluation  Strategic  Plan,  UXO  requirements,  and  more 
specifically,  the  U.S.  Army  requirement  A(1.6a),  titled:  UXO  Screening, 
Detection,  and  Discrimination  and  described  the  FY99  Army  Environmental 
Requirements  and  Technology  Assessments  (AERTA).  This  Army  requirement 
has  been  ranked  as  the  highest  priority  user  need  in  the  Environmental  Cleanup 
Pillar.  In  addition,  this  project  addresses  the  UXO  detection  and  discrimination 
requirements  and  recommendations  described  in  the  Defense  Science  Board  Task 
Force  Final  Report  on  UXO  Clearance  and  Remediation  published  in  1998  and 
will  provide  data  to  support  the  development  of  more  accurate  estimates  of  the 
overall  DOD  UXO  environmental  remediation  costs. 

The  advanced  technologies  demonstrated  as  part  of  this  effort  address  all 
aspects  of  the  requirements  for  land-based,  man  portable  buried  UXO  detection 
and  discrimination  systems.  The  results  of  these  demonstrations  were  used  to 
quantify  the  capability  of  state-of-the-art  systems  to  detect,  locate,  classify,  and 
identify  buried  targets.  The  performance  of  the  advanced  systems  was  compared 
with  the  baseline  comprised  of  traditional  “mag  and  flag”  surveys,  and  costs  and 
production  rates  of  each  technology  were  documented. 

This  technology  demonstration  creates  a  framework  for  the  evaluation  of 
state-of-the  art  sensor  technologies  to  detect,  locate,  and  identify  UXO.  Baseline 
technology  performance  is  established  and  technology  capabilities  and  limita¬ 
tions  are  assessed.  Results  from  this  program  will  be  widely  distributed  to  aid  in 
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the  selection  and  utilization  of  companies,  systems,  and  sensors  for  UXO 
characterization  and  restoration  efforts. 


Objectives  of  the  Demonstration 

The  overall  technical  objective  of  this  demonstration  project  was  to  evaluate 
the  detection  and  discrimination  capabilities  (including  production  rates  and 
costs)  of  advanced  UXO  systems  in  difficult  magnetic  clutter  environments  such 
as  those  encountered  at  Kaho’olawe,  Hawaii.  Three  test  areas  within  JPG  were 
prepared  to  present  a  limited  range  of  conditions  to  the  various  demonstrators  to 
identify  scenarios  where  one  technology  may  be  better  suited  than  the  others. 

The  evaluation  objectives  for  the  demonstrations  were  as  follows: 

a.  To  evaluate  the  demonstrators’  detection  and  discrimination  capabilities 
by  means  of  surveys  of  three  1 -hectare  areas  within  Jefferson  Proving 
Ground  under  realistic  target/  geologic  clutter/  man-made  clutter/ 
topography  scenarios  and  while  operating  as  efficiently  as  possible 
(minimizing  time,  manpower,  and  costs). 

b.  To  evaluate  the  demonstrators’  ability  to  analyze  survey  data  onsite  as 
efficiently  as  possible  and  provide  prioritized  “dig  lists”  with  associated 
classification  confidence  levels.  (These  dig  lists  are  available  via  the  ftp 
server  listed  in  Appendix  B.) 

c.  To  collect  manpower,  time,  and  cost  data  for  all  tasks  required  to 
produce  their  final  products  (prioritized  dig  sheets  and  georeferenced 
anomaly  maps). 

d.  To  compare  the  performance  of  the  advanced  systems  with  the  baseline 
“mag  and  flag”  technology. 

e.  To  provide  high  quality,  ground-truthed,  georeferenced  data  for  post¬ 
demonstration  analysis,  development  of  Receiver  Operating  Charac¬ 
teristic  (ROC)  curves,  and  for  use  by  other  Government,  university, 
and  industry  researchers  to  develop  improved  models  and  analysis 
technologies. 

Regulatory  Issues 

The  primary  regulatory  issue  affecting  UXO  detection  and  discrimination 
technologies  is  gaining  confidence  and  approval  from  Federal,  state,  and  local 
regulators,  stakeholders,  and  users.  In  addition,  acceptance  of  these  innovative 
technologies  from  agencies  such  as  the  U.S.  Army  Corps  of  Engineers  and  the 
Naval  Facilities  and  Engineering  Command  is  needed  to  ensure  that  future 
Requests  for  Proposals  (RFPs)  for  UXO  cleanup  projects  will  be  written  in  a 
manner  that  will  either  sanction  these  technologies,  or  at  least  allow  their 
inclusion  in  proposals  for  site  work. 
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Previous  Testing  of  the  Technology 

Versions  of  the  technologies  demonstrated  under  this  effort  have  been  pre¬ 
viously  tested  as  part  of  the  Defense  Advanced  Research  Projects  Agency 
(DARPA)  Clutter  Experiment  (FY97),  the  Jefferson  Proving  Ground  Phases  II 
through  IV  Demonstrations,  and  a  number  of  ESTCP-funded  field  demonstration 
projects.  However,  this  is  the  first  set  of  controlled  field  experiments  where  the 
three  technologies  have  been  tested  under  realistic  conditions  that  allow  for  side- 
by-side  comparison  of  performance,  production  rates,  and  costs. 
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2  Technology  Description 


Description 

The  three  advanced  technologies  demonstrated  under  this  project  include 
a)  the  Geophex  Ltd.  GEM-3,  a  multichannel  frequency  domain  electromagnetic 
induction  (EMI)  sensor  system  operated  by  Geophex  Ltd.  personnel  with  pro¬ 
cessing  support  from  AETC  Corp.,  b)  the  Naval  Research  Laboratory  (NRL) 
Man-Portable  EM  System  (EMMS)  Adjunct  to  the  Multisensor  Towed  Array 
Detection  System  (MTADS)  system,  a  single  time-channel  time  domain  EMI 
sensor  operated  by  personnel  from  NRL  with  processing  support  from  AETC 
Corp,  and  c)  the  Geonics  Ltd.  EM-63,  a  multi-channel  time  domain  EMI  sensor 
operated  by  personnel  from  NAEVA  Geophysics.  Each  of  the  three  sensors  was 
integrated  into  a  man-portable  platform  that  included  data  acquisition/storage  and 
differential  Global  Positioning  System  (GPS)  receivers.  These  sensor  systems  are 
shown  in  Figures  1  through  3  conducting  surveys  at  the  JPG  test  areas. 
Descriptions  of  these  three  systems  are  included  in  the  Technology 
Demonstration  Plan  (NAVEODTECHDIV  2000a). 

In  addition  to  the  EMI  surveys  conducted  by  these  three  systems,  magnetic 
surveys  of  the  three  areas  were  conducted  by  NRL  with  a  combination  of  the 
MTADS  vehicular-towed  magnetometer  array  and  the  Man-Portable  Magnetom¬ 
eter  System  (MMS)  shown  in  Figures  4  and  5,  and  by  EODT  Technology,  Inc.  (a 
commercial  UXO  services  firm  under  contract  to  the  U.S.  Army  Corps  of  Engi¬ 
neers,  Engineering  and  Support  Center,  Huntsville),  using  the  Schonstedt  hand¬ 
held  GA-52Cx  magnetic  gradiometer  shown  in  Figure  6.  The  MTADS/MMS 
platforms  collected  georeferenced  total  magnetic  field  data  over  the  three  test 
areas.  The  purpose  of  the  MTADS/MMS  survey  was  to  collect  a  more  complete 
data  set  to  support  postdemonstration  analysis  and  to  identify/quantify  any  per¬ 
formance  improvements  resulting  from  adding  magnetometer  information  to  the 
EMI  data.  The  Schonstedt  GA-52  Cx  is  an  analog  magnetic  gradiometer  that 
provides  only  an  audio  signal  to  the  operator  when  it  senses  a  disturbance  in  the 
magnetic  field  (most  likely  caused  by  a  buried  ferrous  object).  The  operator  is 
then  responsible  for  interpreting  the  strength  and  spatial  extent  of  the  audio  signal 
to  determine  if  it  corresponds  to  an  UXO-sized  object;  if  so,  he  places  a  plastic 
pin  flag  at  the  estimated  location  of  the  object.  EODT  personnel  were  provided 
samples  of  emplaced  ordnance  and  were  instructed  to  disregard  any  buried  object 
that  they  determined  to  be  smaller  than  the  smallest  emplaced  munition  (20-mm 
projectiles).  ERDC  personnel  then  surveyed  each  flagged  location  to  produce  the 
georeferenced  “mag  and  flag”  maps  included  in  this  report.  The  purpose  of  the 
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Schonstedt  survey  was  to  establish  a  baseline  for  detection  performance,  cost, 
and  production  rate  for  comparison  with  the  advanced  EMI  systems. 


Strengths,  Advantages,  and  Weaknesses 

The  following  paragraphs  represent  a  summary  of  the  perceived,  claimed, 
and  documented  capabilities  of  each  of  the  three  technology  demonstrators, 
compared  to  currently  fielded,  standard  UXO  technologies  (e.g.,  “mag  and  flag”, 
digital  magnetometer/gradiometer,  and  single-channel  EMI  systems  such  as  the 
EM-61). 


GEM-3 

The  strength  of  the  GEM-3  system  is  claimed  to  lie  in  its  ability  to  rapidly 
collect  multiple  channels  of  complex  frequency  domain  EMI  data  over  a  wide 
range  of  audio  frequencies  (30  Hz  to  over  20  kHz).  This  allows  for  performing 
what  Geophex  Ltd.,  the  developer  of  the  system,  calls  Electromagnetic  Induction 
Spectroscopy  (EMIS)  of  buried  objects  (Won,  Keiswetter,  and  Nobikova  1998). 
EMIS  provides  a  method  to  discriminate  UXO  targets  from  natural  and  manmade 
clutter  objects  by  means  of  their  unique,  complex  (inphase  and  quadrature)  fre¬ 
quency  responses.  The  GEM-3  system  was  the  top  performer  in  the  discrimina¬ 
tion  and  identification  tests  conducted  during  JPG  Phase  IV.  A  concern  and 
possible  weakness  of  the  GEM-3  is  that,  to  accomplish  a  wide  area  detection  and 
discrimination  survey  in  a  reasonable  time,  it  must  keep  the  number  of  frequen¬ 
cies  to  a  relatively  low  number  (seven  in  the  case  of  these  JPG  tests)  and  must 
transmit  them  simultaneously  rather  than  sequentially  as  was  done  during  the 
static  JPG  IV  tests.  This  simultaneous  transmission  of  multiple  frequencies  may 
reduce  the  power  dedicated  to  each  frequency  and  this  could  affect  the  depth 
capability  of  the  GEM-3  system.  Another  possible  weakness  of  the  GEM-3 
system  is  that  high-accuracy  position  information  is  required  to  perform  the 
discrimination.  In  the  previous  JPG  IV  demonstrations,  this  high  degree  of 
position  accuracy  was  obtained  by  means  of  templates  placed  over  specified 
target  locations  allowing  static  point  measurements  to  be  made.  That  approach 
was  not  viable  for  the  wide  area  search  requirements  of  the  current  project  and 
Geophex  had  to  rely  on  Global  Positioning  System  (GPS)  position  information, 
which  results  in  significantly  greater  position  errors  and  sparser  data  sets.  Finally, 
it  was  observed  that  the  GEM-3  system  is  still  in  development  and  the  sensor 
design,  platform,  and  analysis  approaches  have  not  been  optimized  or  finalized. 
This  problem  surfaced  during  the  JPG  field  demonstrations  when  it  was  realized 
that  the  UXO  signatures  library  had  been  collected  with  a  coil  size  that  was 
different  from  the  one  used  for  the  JPG  field  surveys. 
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Figure  2.  EMMS  operated  by  NRL 


Figure  5.  MTADS  operated  by  NRL 


EMMS 


The  EMMS  is  derived  from  the  highly  successful  MTADS  development 
effort  and  thus  incorporates  many  of  its  sensing,  navigation,  and  data  analysis 
system  (DAS)  advances  demonstrated  and  documented  in  a  number  of  ESTCP- 
funded  field  demonstrations.  The  specifications  and  performance  improvements 
incorporated  into  the  version  of  the  EMMS  demonstrated  at  JPG  are  fully 
described  in  the  ESTCP  report  titled  “Man-Portable  Adjuncts  for  the  MTADS” 
(Naval  Research  Laboratory  2001).  Based  on  a  modified  version  of  the  com¬ 
mercially  available  EM-61,  the  most  widely  used  EMI  system  for  UXO  detection 
applications,  the  EMMS  sensor  is  expected  to  have  good  UXO  detection  capa¬ 
bility  to  the  maximum  depths  of  the  objects  emplaced  at  JPG.  Coupled  with  the 
very  high  accuracy  of  the  MTADS-derived,  digital  inclinometer/GPS  system,  the 
EMMS  is  expected  to  produce  high  quality  georeferenced  EMI  data.  A  potential 
limitation  of  the  EMMS  is  the  single  channel  of  data  available,  which  may  limit 
the  discrimination  performance  compared  to  what  can  ultimately  be  achieved  by 
multichannel  systems.  However,  the  extent  to  which  the  additional  data  available 
in  these  other  systems  can  be  exploited  to  improve  performance  has  not  been 
established  in  field  conditions  and  this  demonstration  provides  an  early  oppor¬ 
tunity  to  do  some  initial  comparisons.  In  addition,  the  EMMS  was  used  at  this 
demonstration  with  a  recently  developed  software  classification  algorithm  that 
had  not  been  extensively  evaluated  in  realistic  field  tests. 


EM-63 

The  strengths  of  the  EM-63  are  similar  to  those  attributed  to  the  GEM-3, 
since  they  are  both  capable  of  collecting  multiple  channels  of  information  at  each 
survey  point.  The  EM-63  collects  multiple  channels  of  time  domain  data  for  each 
point  surveyed,  thereby  enhancing  the  amount  of  information  available  to  per¬ 
form  discrimination  and  identification  of  buried  targets.  Unlike  other  prototype 
and/or  developmental  systems,  the  EM-63  is  a  commercially  available  sensor 
(produced  by  Geonics  Ltd.  which  also  manufactures  the  EM-61)  and  has  been 
ruggedized  for  field  use.  Another  significant  strength  of  the  system  demonstrated 
at  JPG  is  the  processing  expertise  of  NAEVA  personnel.  During  previous  JPG 
demonstrations,  NAEVA  has  consistently  ranked  among  the  top  performers,  even 
though  they  had  employed  sensor  data  that  were  equivalent  to  that  of  other  less- 
successful  demonstrators.  Perceived  weaknesses  of  the  NAEVA  EM-63  demon¬ 
stration  system  include  their  limited  experience  with  the  EM-63,  since  it  has  only 
recently  become  available,  and  NAEVA  has  had  very  limited  access  to  the  sensor 
(via  rental  arrangements)  prior  to  the  JPG  tests.  In  addition,  the  analysis  tech¬ 
niques  were  still  under  development  and  had  not  been  fully  tested  nor  imple¬ 
mented  in  transportable  computers  at  the  time  that  NAEVA  arrived  at  JPG  for  the 
required  demonstrations.  Finally,  the  commercial  GPS  system  was  integrated 
with  the  EM-63  only  shortly  before  arriving  at  JPG  and,  as  a  result,  NAEVA 
personnel  had  very  limited  experience  operating  the  system. 
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Factors  Influencing  Cost  and  Performance 

The  factors  evaluated  as  part  of  this  project  that  influence  the  cost  and 
performance  of  each  of  these  systems  include: 

a.  Equipment  setup  and  calibration  time  and  man-hour  requirements. 

b.  Actual  survey  time  and  man-hour  requirements  for  each  of  the  three  test 
areas. 

c.  Downtime  resulting  from  system  malfunctions  and  maintenance 
requirements. 

d.  Reacquisition/resurvey  time  and  man-hour  requirements  (if  any). 

e.  Data  processing/analysis  time  and  man-hour  requirements. 

f.  Detection  performance  and  false  alarm  rates  (as  determined  from 
prioritized  dig  lists  with  associated  confidence  levels). 

g.  .  Discrimination  capability  (ability  to  separate  detected  anomalies  into 

UXO  and  non-UXO  objects). 

h.  Identification  capability  (ability  to  classify  UXO  targets  by  class  (e.g., 
mortar,  projectile)  and  type  (e.g.,  152  mm). 

i.  Predicted  target  location  accuracy  (including  depth  estimates). 

This  demonstration  at  JPG  was  designed  to  collect  the  necessary  information 
to  evaluate  each  of  these  cost  and  performance  factors.  The  Demonstration 
Workplan  (NAVEODTECHDIV  2000a)  includes  a  detailed  description  of  the 
methods  and  metrics  used  to  evaluate  each  of  the  factors. 
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3  Site/Facility  Description 


Background 

The  selection  criteria  for  the  three  JPG  demonstration  areas  are  detailed  in 
the  Site  Preparation  Plan  (NAVEODTECHDIV  2000b).  The  selection  of  the  test 
areas  was  driven  by  the  main  demonstration  objective,  which  was  to  evaluate  the 
performance  of  advanced  EMI  technologies  in  the  presence  of  magnetic  noise 
from  geologic  sources  and  in  different  terrains.  In  addition,  the  three  sites  were 
seeded  with  varying  concentrations  of  inert  UXO  and  man-made  clutter  items. 


Site/Facility  Characteristics 

The  three  1 -hectare  areas  within  JPG  were  selected  to  provide  the  demon¬ 
strators  with  varying  degrees  of  natural  magnetic  clutter  and  terrain  difficulty. 
Area  1  was  selected  because  it  contains  very  high  magnitude  magnetic  anomalies 
from  geologic  sources  that  cover  a  fairly  large  area  as  shown  in  Figure  7.  The 
long  magnetic  anomaly  (red  area)  appearing  near  the  center  of  Area  1  represents 
variations  from  the  background  mean  of +150  nT  to  -100  nT  as  measured  by  the 
MTADS  system  during  previous  JPG  surveys.  Area  1  has  sparse  tree/shrub 
coverage  and  its  topography  includes  rolling  terrain  and  ditches.  Area  1  was 
seeded  with  the  largest  concentration  of  target  and  clutter  items,  and  a  substantial 
number  of  these  were  placed  within  the  high  magnetic  background  locations. 
Area  2,  also  shown  in  Figure  7,  was  chosen  because  it  has  a  significant  number 
of  magnetic  geologic  anomalies  (red  area).  In  Area  2  the  magnetic  anomalies  are 
more  compact  and  lower  in  magnitude  (+35  nT),  thus  providing  a  different 
clutter  problem  from  that  of  Area  1 .  The  topography  in  Area  2  also  includes 
rolling  terrain  and  a  small  ravine.  Area  2  was  seeded  with  a  smaller  number  of 
target  and  clutter  items  than  Area  1.  Area  3  was  chosen  because  it  has  very  low 
amplitude  magnetic  anomalies  from  geologic  sources  and  very  flat  terrain.  This 
area  has  a  variation  from  the  mean  background  of  only  +  6  nT.  Area  3  was 
seeded  with  the  fewest  UXO  target  and  clutter  items.  The  Site  Preparation  Plan 
(Appendix  C)  includes  topographic  maps  of  these  three  areas  prior  to  any  target 
emplacement  activities.  The  emplaced  target  locations  are  included  in  Appen¬ 
dix  C,  Site  Preparation  Plan,  and  are  part  of  the  ground  truth,  which  will  be 
released  at  the  discretion  of  the  ESTCP  Program  Office 
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4  Demonstration  Approach 


Performance  Objectives 

The  rationale  driving  the  design  of  this  demonstration  is  based  on  the  appli¬ 
cation  of  lessons  learned  from  past  JPG  demonstrations  to  address  a  specific 
UXO  Environmental  Remediation  goal  -  the  cleanup  of  Kaho’olawe-type  sites. 
As  a  result,  this  demonstration  was  planned  to  simultaneously  evaluate  the  detec¬ 
tion  and  discrimination  capabilities  of  advanced  UXO  systems  under  realistic 
time  and  cost  (manpower)  considerations.  Based  on  the  results  of  the  JPG  Phase 
III  and  Phase  IV  demonstrations,  we  identified  the  need  to  couple  production 
rates  and  costs  to  the  detection  and  discrimination  capabilities  of  the  systems.  As 
a  result,  this  demonstration  included  survey  and  analysis  time  requirements,  as 
well  as  man-hour  requirements,  in  the  evaluation  factors.  Based  on  concerns 
raised  by  JPG  Phase  IV  demonstrators,  no  unrealistic  (fabricated)  clutter  items 
were  used;  instead,  actual  munitions  fragments  and  magnetic  rocks  from  actual 
UXO  remediation  sites  were  employed.  Also,  based  on  previous  JPG  experience, 
all  inert  UXO  targets  that  had  not  been  previously  fired/shot  were  demagnetized 
prior  to  emplacement  to  simulate  the  magnetic  properties  of  ordnance  that  has 
been  fired.  The  procedures  used  to  demagnetize  the  UXO  targets  are  described  in 
the  Site  Preparation  Plan  and  are  summarized  in  the  following  text.  In  addition, 
extra  precautions  were  taken  to  avoid  surface  disturbances  (e.g.,  the  “bathtub” 
effect)  that  could  alert  the  demonstrators  to  the  presence  of  a  buried  object.  An 
angle  drill/push  rig  was  used  to  emplace  most  of  the  medium  and  large  inert 
UXO  items. 

The  scope  of  this  demonstration  was  not  intended  to  be  a  competition  where 
the  Government  declares  an  overall  winner.  However,  it  was  intended  to  collect 
sufficient  information  from  this  limited  range  of  test  scenarios  to  quantify  the 
advantages  and  disadvantages  of  each  of  the  three  technologies  so  that  they  may 
be  properly  applied  to  specific  UXO  cleanup  problems.  The  immediate  goal  of 
this  effort  is  to  collect  the  data  needed  to  identify  appropriate  technologies  to 
transition  to  Kaho’olawe-type  environments  where  natural  (magnetic  rocks/soils) 
and  man-made  (munitions  fragments)  clutter  have  rendered  cleanup  operations 
using  conventional  technologies  both  expensive  and  ineffective.  A  longer-term 
objective  of  this  demonstration  is  to  provide  high-quality,  georeferenced  data  to 
support  sensor  development  and  improvements  in  UXO  analysis  technologies. 

The  goals  of  this  demonstration  are  to: 
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a.  Evaluate  demonstrators  operating  in  realistic  target,  geologic  clutter, 
man-made  clutter,  and  topography  scenarios.  Criteria  for  evaluation  are: 

(1)  Detection,  discrimination,  and  identification  capabilities  based  on 
prioritized  dig  lists  produced  from  onsite  data  analysis. 

(2)  Manpower,  time,  and  costs  required  to  produce  onsite  dig  lists. 

(3)  Additional  detection  and  discrimination  capabilities  based  on 
offsite,  post  demonstration  analyses. 

b.  Provide  baseline  data  for  comparison  of  advanced  EMI  technologies  with 
traditional  “mag  and  flag.” 

c.  Archive  high-quality,  ground-truthed,  georeferenced  data  for  broader  use 
in  the  UXO  technology  development  community. 

Physical  Setup  and  Operation 

Descriptions  of  the  inert  UXO  targets  and  the  clutter  items  used  for  this 
demonstration  are  included  in  the  Site  Preparation  Plan  (Appendix  C).  Photo¬ 
graphs,  descriptions,  dimensions,  and  emplacement  information  of  each  target 
and  clutter  item  are  available  as  part  of  the  ground-truthed  information  in  CD 
form  from  the  ESTCP  program  office.  Briefly,  the  UXO  targets  ranged  from 
20-mm  projectiles  buried  near  the  surface  to  155-mm  projectiles  buried  up  to 
1 .2  m  below  the  surface.  Clutter  items  emplaced  ranged  from  small  (less  than 
0.5-kg)  to  large  (up  to  5-kg)  munitions  fragments  and  included  large  magnetic 
rocks  and  man-made  clutter  such  as  horseshoes  and  metal  banding.  A  2-m  by 
2-m  area  around  each  planned  target  location  was  surveyed  with  a  G-858 
magnetometer  to  detect  and  remove  any  metallic  objects  prior  to  emplacing  an 
inert  UXO  target.  The  results  of  these  surveys  are  available  from  the  data  server 
listed  in  Appendix  B. 

Degaussing  of  the  small  UXO  items  was  performed  using  an  AudioLab 
magnetic  tape  degausser,  Model  TD-5,  and  the  magnetic  signatures  were  mea¬ 
sured  with  a  Geometries  Model  G-822L  portable  magnetometer.  Prior  to  degaus¬ 
sing,  the  magnetic  state  of  the  item  was  determined  by  aligning  it  with  the  nose 
pointing  North  at  a  set  distance  from  the  magnetometer.  The  item  was  then 
rotated  180  deg  so  that  the  tail  was  facing  the  magnetometer  and  at  the  same 
preset  distance.  The  difference  between  the  two  readings  was  recorded  and  the 
item  placed  on  the  degausser  to  lower  the  difference  to  2nT  or  20-percent  of  the 
original  difference  reading  with  the  degausser.  The  time  of  degaussing  and  the 
power  setting  used  were  varied  as  needed  to  obtain  the  desired  reduction  in  the 
difference  measurements.  The  larger  items,  such  as  5-in.  projectiles  and  155-mm 
projectiles,  were  degaussed  using  the  fixed  facilities  atNAVEODTECHDIV. 

The  magnetization  of  each  large  UXO  item  was  checked  in  the  three  orthogonal 
orientations  using  the  same  magnetometer  and  procedures  as  before,  and  the  item 
was  then  degaussed  in  the  three  directions  until  all  three  difference  readings 
matched  the  tolerances  listed  previously. 

Samples  of  each  of  the  UXO  targets  emplaced  were  made  available  to  each 
demonstrator  prior  to  arriving  on  site  for  signature  collection  and  system  training, 
and  additional  samples  were  also  available  at  the  demonstration  site  for 
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calibration  purposes.  Unlike  the  previous  JPG  tests,  the  clutter  items  were  not 
made  available  to  the  demonstrators  for  signature  collection  and  system  training. 
A  2  m-long  by  0.75  m-wide  by  0.75  m-deep  trench  in  the  calibration  area 
(NAVEODTECHDIV  2000a)  located  near  test  area  2  was  made  available  to 
demonstrators  for  system  calibration  and  checkout  purposes. 

Three  resurveyed,  first-order  control  points  located  within  the  original  JPG 
40  acre  site  were  made  available  for  demonstrators  to  set  up  GPS  base  stations. 
The  primary  reference  monument  is  located  near  the  southwest  comer  of  Test 
Area  3  and  was  used  as  the  reference  point  for  all  site  preparation  and  demon¬ 
stration  activities.  This  marker  was  brought  up  to  first-order  accuracy  during  the 
site  preparation  activities,  and  updated  coordinates  were  provided  to  the  demon¬ 
strators  prior  to  the  scheduled  demonstrations.  Two  other  monuments  were  also 
resurveyed  to  first-order  accuracy  and  made  available  to  the  demonstrators.  One 
was  designated  Monument  #1  (see  NAVEODTECHDIV  2000a)  and  is  located 
within  test  Area  2  near  its  south  boundary.  The  other  is  designated  Monument  #3 
and  is  located  approximately  40  m  southwest  of  Test  Area  1 . 

The  Government  installed  a  meteorological  station  near  the  three  1 -hectare 
areas  to  continuously  record  weather  conditions  beginning  1  month  prior  to  the 
demonstrations  and  continuing  through  the  end  of  the  testing  periods.  These  data 
were  automatically  recorded  at  15-min  intervals  and  transmitted  to  ERDC  for 
archiving.  It  includes  wind  speed,  wind  direction,  solar  insolation,  rainfall,  temp¬ 
erature,  humidity,  and  soil  moisture.  The  archived  data  are  available  from  ERDC 
upon  request.  It  should  be  noted  that  other  than  short  delays  (under  2-hr  duration) 
resulting  from  rain,  there  were  no  significant  weather  events  during  the  scheduled 
field  work  that  affected  the  demonstrations.  There  were,  however,  several  severe 
weather  events  recorded  during  nonsurvey  periods  that  resulted  in  partial  flood¬ 
ing  of  some  of  the  test  areas  and  forced  the  demonstrators  to  modify  their 
planned  survey  order  to  allow  time  for  the  areas  to  drain.  Some  small  portions  of 
the  test  grids  remained  under  standing  water  for  long  periods  of  time,  and  demon¬ 
strators  had  to  survey  in  conditions  characterized  by  10  to  20  cm  of  standing 
water  and  soft  soil. 

The  four  comers  of  each  test  area  were  surveyed  by  the  Government  and 
marked  with  a  metallic  marker  (rebar)  driven  flush  with  the  ground  for  use  by  the 
demonstrators  as  fiducial  markers  to  check/correct  their  position  information. 
Plastic  pin  flags  were  placed  at  5-m  increments  along  the  perimeter  of  each  of  the 
test  areas  to  assist  in  maintaining  proper  lane  spacing. 

The  demonstration  test  areas  were  mowed  as  part  of  the  site  preparation 
activities  during  June  2000.  Prior  to  starting  surveys,  the  first  demonstrator 
(Geophex)  inspected  the  test  areas  and  determined  that  additional  mowing  was 
not  required  for  their  survey  activities.  The  site  was  mowed  for  a  second  time 
prior  to  arrival  of  the  second  demonstrator  (NRL).  No  additional  mowing  was 
conducted  until  the  completion  of  the  “mag  and  flag”  surveys  during 
November  2000. 
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Sampling  Procedures 

The  Demonstration  Work  Plan  describes  the  procedures  required  for  each  of 
the  demonstrations.  Demonstrators  were  responsible  for  developing  their  specific 
survey  plans  (including  lane  spacing,  sampling  rate,  number  of  channels 
recorded,  calibration  methods,  etc.)  and  these  procedures,  together  with  their 
analysis  techniques,  are  described  in  Appendix  C. 

Each  of  the  demonstrators  was  allotted  one  10-day  period  (Monday  through 
Wednesday  of  the  following  week)  during  14  August  2000  through  20  September 
2000  to  complete  their  surveys  and  submit  the  required  onsite  dig  sheets.  Each 
workday  could  extend  to  a  maximum  of  10  hr  onsite. 


Analytical  Procedures 

The  evaluation  factors,  metrics,  products,  and  procedures  related  to  this 
demonstration  are  described  in  the  Demonstration  Workplan  and  include  the 
following  information: 

a.  Equipment  setup,  calibration  time,  and  man-hour  requirements. 

b.  Actual  survey  time  and  man-hour  requirements  for  each  of  the  three  test 
areas. 

c.  Downtime  because  of  system  malfunctions  and  maintenance 
requirements. 

d.  Reacquisition/resurvey  time  and  man-hour  requirements  (if  any). 

e.  Actual  data  processing/analysis  time  and  man-hour  requirements  (all  to 
be  performed  onsite). 

f  Prioritized  dig  lists  with  associated  confidence  levels. 

g.  Discrimination  capability  (ability  to  separate  detected  anomalies  into 
UXO  and  non-UXO  objects). 

h.  Identification  capability  (ability  to  classify  UXO  targets  by  class  (e.g., 
mortar,  projectile)  and  type  (e.g.,  152  mm). 

i.  Predicted  target  location  accuracy  (including  depth  estimates). 

j.  Georeferenced  anomaly  maps. 

k.  Probabilities  of  Detection  (Pd). 

l.  False  Alarm  Rates  (FAR). 

The  method  for  determining  and  documenting  the  first  three  items  involved 
the  Government  onsite  representatives  tracking  and  recording  the  number  of 
personnel  and  time  spent  performing  each  of  the  tasks.  Adequate  rest  and  lunch/ 
dinner  breaks  were  provided  and  these  times  were  not  included  in  the  perform¬ 
ance  metrics  calculations.  If,  during  the  analysis  of  the  data,  the  demonstrator 
determined  that  he  needed  to  resurvey  any  part  of  the  test  areas  or  any  previously 
detected  anomalies,  all  setup,  calibration,  survey,  downtime,  and  reacquisition 
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times  and  man-hour  requirements  were  recorded  individually  (as  in  items  a 
through  c),  but  were  compiled  separately  as  reacquisition/resurvey  time  (item  d). 

To  evaluate  item  e,  the  Government  required  that  all  data  processing  and 
analysis  tasks  required  to  produce  items  f  through  j  be  conducted  in  the  JPG 
office  trailer,  and  no  data  be  taken  offsite  until  these  items  are  submitted  to  the 
onsite  Government  representative.  Demonstrators  were  responsible  for  providing 
all  computer  hardware,  software,  and  support  equipment  needed  to  produce  the 
required  analysis  products. 

Development  and  evaluation  of  (previously  listed)  items  f  through  j  are  as 
follows: 

a.  Each  demonstrator  was  required  to  combine  the  EM  sensor  data  with  the 
GPS  position  information  to  develop  two-dimensional  (2-D)  anomaly 
maps  of  each  1 -hectare  area.  These  anomaly  maps,  together  with  the 
corresponding  digital  geophysical  sensor  data,  were  then  analyzed  to 
identify  all  detected  anomalies  that  could  potentially  be  a  buried  UXO 
target  for  each  of  the  three  areas.  These  anomalies  were  then  tabulated 
into  one  preliminary  dig  sheet  for  each  test  area.  The  objective  of  this 
phase  was  to  include  as  many  anomalies  in  these  lists  as  required  to 
ensure  as  high  a  Pd  as  possible  for  the  full  range  of  UXO  targets 
considered. 

b.  Each  anomaly  in  each  list  was  then  further  analyzed  to  develop  the  final 
prioritized  dig  sheets  as  illustrated  in  Table  1.  The  demonstrators  were 
asked  to  refine  the  location  (x,  y)  and  estimate  the  burial  depth  (z)  of 
each  object,  to  attempt  to  separate  (discriminate)  UXO  from  clutter 
items,  to  identify  UXO  by  class  and  type  (if  possible),  and  to  rank  the  list 
in  the  following  descending  order:  UXO  -  high  confidence,  UXO  - 
medium  confidence,  UXO  —  low  confidence,  Clutter  —  low  confidence, 
Clutter  —  medium  confidence,  and  Clutter  —  high  confidence.  In  addition, 
the  list  was  required  to  include  predicted  ordnance  class  and  size  (e.g., 
mortar/8 1  mm)  for  all  anomalies  declared  as  UXO  with  high  and 
medium  confidence  levels,  and,  if  possible,  UXO  orientation  (Azimuth 
and  Inclination). 


Table  1 

Sample  Dig  Sheet 

Sample  C 

>ig  List  DIG  LU 

5T:  J_  Demon 

strator:  EMMS  Test  Area:  1  Including  20  mm  ?:  NO  || 

Ranking 

Northing 

meters 

Easting 

meters 

Depth 

meters 

Type 

ordnance/ 

clutter 

Confidence 

Size/W 

eight 

Azimuth 

degrees 

Inclination 

degrees 

Class 

001 

4309738.557 

641594.2038 

0.9144 

ordnance 

mm 

Large 

180 

20 

projectile 

152  mm 

050 

4309689.964 

641519.4151 

0.89042 

ordnance 

low 

small 

_ 

_ 

projectile 

unknown 

165 

4309700.031 

641516.8877 

0.82296 

clutter 

medium 

- 

- 

USSR 

- 
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c.  Each  demonstrator  was  then  required  to  specify  a  threshold  (row)  on 
each  prioritized  list  where  he  would  recommend  that  all  objects  at  or 
above  that  threshold  be  excavated  and  those  below  be  left  in  place.  The 
goal  of  this  step  is  to  evaluate  the  demonstrators’  ability  to  discriminate 
UXO  targets  from  clutter.  To  add  realism  to  this  discrimination  decision 
process,  demonstrators  were  instructed  beforehand  that  the  following 
cost  factors  would  be  applied:  1)  For  every  clutter  item  selected  for 
“digging,”  a  $200  cost  penalty  was  assigned  (the  average  cost  of  exca¬ 
vating  items  at  actual  UXO  remediation  sites).  2)  To  reflect  the  unaccept¬ 
able  risk  of  leaving  UXO  in  the  ground,  a  very  high  penalty  was  assigned 
if  any  detected  anomaly  that  corresponded  to  a  UXO  target  was  errone¬ 
ously  declared  as  clutter  and  placed  in  the  “no  dig”  portion  of  the  list.  As 
a  result,  if  one  or  more  UXO  items  were  placed  in  the  “no  dig”  portion  of 
the  list,  it  would  be  assumed  that  the  grid  (i.e.,  the  entire  1 -hectare  area) 
has  failed  the  Quality  Assurance  and/or  regulatory  acceptance  and  a  cost 
penalty  equal  to  the  cost  of  a  resurvey  would  be  assigned.  One  or  more 
missed  targets  (anomalies  too  weak  to  be  included  in  the  lists  developed 
in  the  description  of  equipment  setup,  calibration  time,  and  man-hour 
requirements  discussed  previously)  in  each  area  were  also  assigned  a  cost 
factor  equal  to  the  cost  of  a  resurvey,  but  it  should  be  noted  that  they 
reflect  a  deficiency  in  the  sensor  rather  than  in  the  analysis  and  decision 
making  process.  Missed  targets  are  also  reflected  in  the  less  than 
100-percent  maximum  Pd  achieved  by  each  system  and  are  documented 
in  this  report  to  aid  regulators  and  managers  in  assessing  residual  risks 
associated  with  the  various  sensing  technologies. 

Items  k  and  1  were  calculated  from  the  prioritized  dig  lists  as  follows:  Maxi¬ 
mum  achievable  Pds  for  each  area  were  calculated  as  the  number  of  items  in  the 
entire  list  that  correspond  to  emplaced  UXO  targets  (even  though  they  may  have 
been  misclassified  as  clutter)  divided  by  the  actual  number  of  UXO  targets 
emplaced  in  that  site.  Note  that  in  order  to  be  declared  a  correct  detection,  the 
declared  object  location  must  be  within  a  1-m  radius  of  the  actual  emplaced 
target  location.  The  operating  (single-point)  Pd  was  determined  by  calculating 
the  number  of  actual  UXO  targets  that  are  included  in  the  list  at  or  above  the 
threshold  described  in  the  previous  paragraph.  Similarly,  the  operating  (single¬ 
point)  FAR  was  calculated  as  the  number  of  clutter  items  that  are  above  the  dig 
threshold.  An  ROC-like  curve  was  developed  by  the  Government  by  varying  the 
dig  threshold  until  the  maximum  Pd  was  reached  and  computing  Pd  and  FAR  at 
each  increment.  Performance  comparisons  between  systems  include  using  the 
ROC-like  curves  to  determine  FAR  at  the  Pd  required  for  Kaho’olawe  Tier  II 
clearance  (Pd  =  85-percent)  and  also  using  the  single  point  performance  (Pd  and 
FAR)  of  the  mag  and  flag  surveys  as  a  baseline. 

After  each  demonstrator  had  submitted  the  dig  sheets  described  above,  the 
timing  for  the  analysis  tasks  was  stopped  and  he  was  to  be  given  the  opportunity 
to  reanalyze  the  data  to  develop  prioritized  dig  sheets  that  take  into  account  only 
targets  larger  than  20-mm  projectiles  (20-mm  projectiles  were  assumed  to  be 
clutter  for  this  portion  of  the  evaluation).  These  dig  sheets  were  to  be  submitted 
to  the  Government  representative  prior  to  leaving  the  JPG  site.  However,  because 
of  a  late  start  and  ensuing  hardware  problems  that  required  additional  delays  for 
collection  of  additional  calibration,  the  first  onsite  demonstrator  (Geophex  Ltd.) 
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was  unable  to  complete  all  of  the  required  data  analysis  in  the  allotted  time  They 
requested,  and  ESTCP  approved,  a  deviation  from  the  Workplan  requirement  that 
all  processing  be  conducted  onsite.  As  a  result,  only  the  initial  sets  of  dig  sheets 
(EM  only  including  all  targets)  were  required  to  be  submitted  prior  to  departing 
the  site.  In  addition,  the  last  demonstrator  (NAEVA)  did  not  deploy  their  com¬ 
puter  workstations  to  the  JPG  site,  and  subsequently  requested  and  received 
approval  from  the  ESTCP  office  to  perform  the  processing  offsite.  As  a  result, 
the  integrity  of  the  onsite  analysis  costs  was  compromised  and  affected  the 
overall  cost  evaluation  included  in  Section  6  of  this  report.  NAEVA  transmitted 
the  field  survey  data  offsite  for  processing  and  was  able  to  submit  a  set  of  dig 
sheets  prior  to  departing  the  site  so  that  a  comparison  of  the  detection  and 
discrimination  performance  is  still  viable. 

After  all  onsite  analysis  products  had  been  submitted,  the  demonstrators  were 
provided  with  magnetometer  data  collected  by  MTADS.  Demonstrators  were 
then  requested  to  reanalyze  their  data  offsite  using  this  additional  information  to 
develop  final  prioritized  dig  sheets  for  each  test  area. 
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Performance  Data 

In  accordance  with  the  Demonstration  Test  Plan,  each  of  the  demonstrators 
was  responsible  for  determining  the  best  method  of  employing  his  system  in 
order  to:  (a)  ensure  full  coverage  of  each  test  area,  (b)  collect  high-quality  sensor 
data  to  support  detection  and  discrimination  requirements,  (c)  achieve  high  pro¬ 
duction  rates,  and  (d)  minimize  man-hour  requirements  and  costs.  All  demon¬ 
strators  were  able  to  complete  the  field  surveys  within  the  allotted  time  periods 
(NAVEODTECHDIV  2000a).  Figures  8  through  16  show  the  georeferenced 
anomaly  maps  produced  by  each  of  the  systems  used  during  these  demonstra¬ 
tions.  The  Geophex  maps  depict  the  930-Hz  quadrature-phase  data  in  parts-per- 
million  referenced  to  the  primary  field  strength  and  color-coded  as  shown  in  the 
corresponding  color  scale.  The  EMMS  maps  depict  the  single-channel  EMI 
sensor  data  in  millivolts  and  color-coded  as  indicated  in  the  corresponding  false 
color  scale.  The  NAEVA  maps  depict  the  EM-63  time  gate  number  10  readings 
in  millivolts,  color-coded  as  indicated  in  the  corresponding  scale.  In  view  of  the 
fact  that  each  demonstrator  included  different  data  (and  in  some  cases  only  a 
small  subset  of  the  sensor  data  acquired)  in  these  anomaly  maps,  direct  compari¬ 
son  and  evaluation  of  the  quality  and  utility  of  each  of  these  maps  is  inappropri¬ 
ate.  The  only  overall  conclusions  that  may  be  derived  from  these  maps  is  that  all 
three  systems  demonstrated  the  capability  to  suppress  the  high  magnetic  back¬ 
ground  from  geologic  sources  and  that  all  demonstrated  the  capability  to  provide 
high-quality  (well-localized,  high  signal-to-background  target  signatures)  geo¬ 
referenced  data. 

The  results  of  the  “mag  and  flag”  surveys  conducted  by  EODT  (a  commer¬ 
cial  UXO  firm  under  contract  to  the  Corps  of  Engineers,  Huntsville  Engineering 
Center)  were  used  as  a  baseline  for  documenting  performance  and  cost  improve¬ 
ments  from  the  application  of  advanced  EMI  technologies.  The  locations  flagged 
by  EODT  were  surveyed  by  Government  personnel  using  Kinematic  Differential 
GPS  (KDGPS)  equipment,  and  the  results  are  presented  in  Figures  17  and  18. 
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Figure  10.  NAEVA  EM-63  survey  results  of  Area  1 
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Figure  11.  Geophex  Ltd.  GEM-3  survey  of  Area  2 
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Figure  1 3.  NAEVA  EM-63  survey  of  Area  2 
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Figure  16.  NAEVA  EM-63  survey  of  Area  3 
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Data  Assessment 


The  demonstrators’  analyses  of  the  field  survey  data  were  performed  in  three 
stages.  The  data  were  initially  analyzed  prior  to  leaving  the  JPG  test  site,  in  order 
to  produce  three  prioritized  dig  lists  (one  for  each  test  area)  that  contained  all 
anomalies  investigated.  The  data  were  also  analyzed  offsite  to  produce  three 
additional  prioritized  dig  lists  that  included  only  those  targets  estimated  to  be 
larger  than  a  20-mm  projectile  (i.e.,  the  smallest  object  of  interest  corresponded 
to  a  57-mm  projectile).  Lastly,  after  submission  of  the  second  set  of  dig  lists,  the 
demonstrators  were  provided  with  the  MTADS  mag  data  and  each  was  requested 
to  submit  two  additional  sets  of  prioritized  dig  lists  (with  and  without  20  mm) 
that  included  combined  mag  and  EMI  analysis  results.  These  dig  lists  are  avail¬ 
able  via  the  ftp  server  listed  in  Appendix  B.  It  should  be  noted  that  the  demon¬ 
strators,  as  well  as  other  researchers  (e.g.,  U-Hunter,  AETC,  Duke  University), 
conducted  additional  analyses  of  these  datasets.  Results  from  these  analyses  are 
not  included  in  this  report  but  will  be  evaluated  and  documented  in  subsequent 
reports. 

A  number  of  postdemonstration  adjustments  to  the  ground  truth  were  neces¬ 
sary  in  order  to  accurately  account  for  anomalies  resulting  from  metallic  objects 
that  were  neither  detected  nor  emplaced  during  the  site  preparation  for  this 
demonstration.  After  initial  evaluation  of  the  submitted  dig  lists,  it  became 
apparent  that  all  demonstrators  declared  targets  at  locations  where  no  items  had 
been  emplaced  and  where  the  magnetic  anomalies  (from  geologic  sources)  were 
not  significant.  A  decision  was  made  to  excavate  those  locations  within  the  three 
test  areas  where  two  or  more  demonstrators  had  declared  UXO  targets.  The 
digging  revealed  that  in  Area  3  (the  site  north  of  the  former  40-acre  site  during 
prior  JPG  demonstrations)  all  of  the  declarations  corresponded  to  farm-related 
ferrous  objects  such  as  portions  of  horseshoes,  plow  points,  and  harness  hard¬ 
ware,  and  did  not  include  any  items  from  previous  JPG  demonstrations.  As  a 
result,  it  was  decided  to  include  all  of  declarations  not  corresponding  to  emplaced 
items  in  Area  3  as  false  alarms  due  to  non-UXO  ferrous  objects. 

On  the  other  hand,  in  Areas  1  and  2  (which  are  inside  of  the  40-acre  site),  the 
limited  digging  revealed  a  number  of  inert  UXO  left  from  previous  JPG  demon¬ 
strations  including  inert  projectiles,  mortars,  flares,  and  fabricated  clutter  items. 
Even  though  the  JPG  IV  ground  truth  had  been  used  to  clear  these  areas,  it 
became  obvious  that  items  emplaced  during  earlier  demonstrations  had  remained. 
As  a  result,  the  Government  examined  the  JPG  I  -  m  ground  truth  and  identified 
items  that  matched  the  locations  of  anomalies  declared  by  any  one  of  the  demon¬ 
strators  as  UXO  targets.  These  objects  were  then  removed  from  the  evaluation  of 
results.  All  other  UXO  target  declarations  that  did  not  correspond  to  items  that 
were  emplaced  as  part  of  this  ESTCP  project,  and  which  were  not  included  in  the 
ground  truth  from  the  prior  JPG  demonstrations,  were  evaluated  as  false  alarms. 
The  option  of  limiting  the  evaluation  to  only  the  objects  emplaced  for  this  demo¬ 
nstration  was  considered  and  rejected  because  it  would  defeat  the  primary  objec¬ 
tive  of  the  test,  which  was  to  evaluate  system  performance  in  high  natural 
magnetic  background  environments. 
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Assessment  of  detection  performance 

One  of  the  critical  evaluation  factors  for  this  demonstration  is  the  detection 
performance  of  the  advanced  systems.  The  metrics  used  to  quantify  the  detection 
performance  consist  of  the  pseudo  ROC  curves,  the  single-point  Pd/FAR,  and  the 
maximum  achievable  Pd.  The  methods  used  to  estimate  these  metrics  from  the 
prioritized  dig  lists  are  described  in  detail  in  Chapter  4.  Briefly,  the  pseudo  ROC 
curve,  which  graphically  represents  the  target  detection  percentage  vs  the  number 
of  false  alarms  (or  false  alarm  rate  in  number  of  false  alarms  per  hectare),  is  cal¬ 
culated  by  sequentially  moving  from  the  top  of  the  prioritized  dig  list  (i.e.,  the 
highest  confidence  UXO  target  declaration)  and  determining  if  each  object  on  the 
list  (whether  classified  as  target  or  clutter)  corresponds  to  an  emplaced  target 
location  (a  detection)  or  not  (a  false  alarm).  The  single-point  Pd/FAR  perform¬ 
ance  is  based  on  the  point  on  the  ROC  curve  that  corresponds  to  the  contractor- 
specified  dig  point  on  the  prioritized  dig  list,  and  the  maximum  achievable  Pd  is 
based  on  the  highest  point  on  the  ROC  curve.  These  performance  metrics  are 
presented  in  the  following  graphs.  The  single-point  Pd/FAR  rate  is  shown  as  a 
colored  triangle  on  the  ROC  curve,  and  the  green  diamond  corresponds  to  the 
single-point  Pd/FAR  performance  point  of  the  “mag  and  flag”  survey. 

There  are  several  points  to  keep  in  mind  when  interpreting  these  pseudo 
ROC  curves:  (a)  the  abscissa  in  the  pseudo  ROC  curves  is  not  Pfa  but  rather  total 
number  of  false  alarms  or,  equivalently,  FAR  (number  of  false  alarms  per  hec¬ 
tare).  As  a  result,  the  absolute  slope  of  the  curve  has  no  intrinsic  meaning,  but  is 
nevertheless  useful  for  comparing  relative  performance  between  different 
systems,  (b)  These  curves  combine  detection  and  discrimination  of  ordnance 
from  nonordnance.  Thus,  the  initial  pseudo  ROC  curve’s  slope  represents  the 
anomalies  that  the  demonstrator  has  declared  as  UXO  with  the  highest  confi¬ 
dence;  a  flat  slope  in  this  area  would  indicate  very  poor  discrimination  capability, 
(c)  Similarly,  the  final  slope  of  the  pseudo  curve  represents  anomalies  that  the 
demonstrator  has  declared  as  clutter  with  high  confidence;  a  positive  slope  in  this 
area  indicates  that  there  are  UXO  targets  which  the  demonstrator  would  leave 
unexcavated. 

Figure  19  shows  the  detection  performance  of  the  three  demonstrators  based 
on  the  results  of  the  onsite  analysis  that  included  all  potential  targets.  The  red 
traces  show  the  performance  results  of  the  Geophex  Ltd.  GEM-3  system.  The 
relatively  flat  slopes  of  these  ROC  curves  indicate  that  the  analysis  performed  on 
the  GEM-3  data  were  not  effective  in  discriminating  UXO  targets  from  clutter. 
The  Pd  performance  of  the  GEM-3  was  superior  to  that  of  the  standard  “mag  and 
flag”  in  the  more  difficult  magnetic  clutter  environments  of  Areas  1  and  2  but  did 
not  demonstrate  enhanced  capability  it  in  the  low-noise  environment  of  Area  3. 

In  Areas  1  and  2,  the  single  point  GEM-3  Pd/FAR  performance  failed  to  meet  the 
85-percent  specified  to  meet  the  Kaho’olawe  Tier  n  requirements.  The  GEM-3 
achieved  100-percent  detection  at  all  three  sites  but  only  at  the  expense  of  a 
significant  number  of  false  alarms. 

In  Figure  19,  the  blue  traces  show  the  performance  results  for  the  NRL 
EMMS.  The  steep  early  slope  of  the  ROC  curves  indicates  significant  discrimi¬ 
nation  capability.  The  EMMS  outperformed  the  “mag  and  flag”  system  at  all 
three  test  areas,  and  the  single-point  performance  points  met  the  Kaho’olawe 


32 


Chapter  5  Performance  Assessment 


Tier  II  requirements.  Based  on  the  maximum  of  the  ROC  curves,  the  EMMS  did 
not  achieve  100-percent  detection  at  any  of  the  three  sites. 

In  Figure  19,  the  yellow  traces  show  the  corresponding  performance  results 
for  the  NAEVA  EM-63  system.  Again,  the  steep  initial  slopes  of  the  ROC  curves 
indicate  significant  discrimination  capability.  The  Pd  performance  of  the 
NAEVA  system  was  significantly  better  than  “mag  and  flag”  across  all  sites. 
NAEVA’ s  ROC-based  performance  was  very  similar  across  Areas  1  and  3  and 
considerably  lower  for  Area  2.  The  single-point  performance  points  meet  the 
Kaho’olawe  requirements.  The  EM-63  system  did  not  achieve  100-percent  detec¬ 
tion  at  any  of  the  three  sites. 

The  naturally  occurring  geologic  magnetic  noise  and  the  emplaced  magnetic 
rocks  presented  no  problems  to  the  three  EMI  systems.  All  false  alarms  included 
in  the  submitted  dig  lists  are  attributable  to  metallic  clutter,  and  analyses  of  the 
georeferenced  maps  show  no  discernible  anomaly  over  any  of  the  emplaced 
magnetic  rocks.  Overall,  NRL  and  NAEVA  demonstrated  similar  discrimination 
and  false  alarm  rate  performance  and  both  were  significantly  higher  than  the 
demonstrated  performance  of  Geophex  Ltd.  NRL’s  ROC  performance  for  Area  2 
was  slightly  better  than  NAEVA’s,  while  NAEVA’s  was  very  slightly  better  for 
Area  1.  Overall,  Geophex  was  the  only  system  that  demonstrated  100-percent  Pd 
at  any  of  the  three  sites. 

Figure  20  shows  the  detection  performance  of  the  three  demonstrators  based 
on  the  results  of  the  offsite  analyses  that  excluded  objects  that  were  estimated  to 
be  the  size  of  20  mm  projectiles  or  smaller.  The  objective  of  this  analysis  was  to 
determine  the  system  performance  based  on  the  more  commonly  encountered, 
midsized  (57  mm  and  larger)  UXO  targets.  It  should  be  noted  that  no  compari¬ 
sons  with  “mag  and  flag”  results  are  included  in  these  figures  because  analog 
magnetometers  lack  the  capability  to  record  the  sensor  data  for  reanalysis. 

In  Figure  20,  the  red  traces  indicate  that  the  ROC-based  performance  of  the 
GEM-3  system  improved  considerably  from  the  onsite  results  shown  previously 
in  Figure  19.  The  offsite  ROC  curves  have  significantly  steeper  slopes  (for  all 
three  areas)  indicating  much  improved  false  alarm  reduction  capability.  The 
operating  Pd/FAR  points,  however,  are  much  lower  than  in  the  previous  set  and, 
as  a  result,  the  GEM-3  operating  Pd  was  below  80  percent  and  failed  to  meet 
Tier  II  requirements  for  all  three  areas.  The  GEM-3  achieved  a  max  Pd  of 
100  percent  only  on  Area  1  and  achieved  only  81 -percent  max  Pd  in  Area  3.  The 
significant  decrease  in  operating  and  max  Pds  from  the  earlier  results  (where 
100-percent  max  Pd  was  achieved  at  all  three  areas)  is  difficult  to  explain.  The 
objects  dropped  from  the  earlier  dig  lists  as  a  result  of  this  analysis  consisted  of  a 
105-mm  projectile  in  Area  2  and  a  60-mm  mortar  and  a  76-mm  projectile  in  Area 
3  with  an  81 -mm  mortar. 

In  Figure  20,  the  blue  traces  show  the  corresponding  performance  for  the 
NRL  EMMS  system.  Again,  comparison  of  these  results  with  those  provided 
onsite  indicates  significant  improvement  in  ROC  curve-based  performance.  In 
addition,  the  EMMS  operating  Pd/FAR  points  improved  substantially,  particu¬ 
larly  in  Area  1  where  100-percent  Pd  was  obtained  with  only  55  false  alarms. 
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Figure  19.  Detection  performance  of  GEM-3,  EMMS,  EM-63  system  (onsite 
results) 
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Figure  20.  Detection  performance  of  GEM-3,  EMMS,  EM-63  system  (w/o  20-mm 
targets) 
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Maximum  Pd  also  increased  slightly  in  the  other  two  areas  and  exceeded  the 
Tier  II  requirements. 

In  Figure  20,  the  yellow  traces  show  the  corresponding  performance  for  the 
NAEVA  EM-63  system.  Again,  a  comparison  with  earlier  results  shows 
improvement  in  all  factors.  By  far  the  greatest  improvement  is  seen  in  Area  2 
where  the  operating  Pd  increased  from  85  to  92  percent,  and  the  corresponding 
false  alarms  were  reduced  from  128  to  56.  Max  Pd  increased  significantly  in 
Areas  1  and  2,  slightly  in  Area  3,  and  exceeded  Tier  II  requirements,  but  again, 
NAEVA  failed  to  reach  the  100-percent  max  Pd  in  all  areas. 

Overall,  NRL  and  NAEVA  demonstrated  similar  ROC-based  performance, 
which  was  again  significantly  better  than  that  demonstrated  by  Geophex.  In 
addition,  both  NRL  and  NAEVA  significantly  improved  their  dig  point  selection, 
since  nearly  everything  beyond  this  point  is  nonordnance.  NRL  demonstrated 
slightly  higher  performance  in  Areas  1  and  3,  while  NAEVA  was  slightly  higher 
in  Area  2.  It  can  be  concluded  that  excluding  the  small  targets  resulted  in 
significant  performance  improvements  for  the  EMMS  and  EM-63  systems. 

Figure  21  shows  the  performance  of  the  three  demonstrators  when  the 
MTADS  mag  data  were  added  to  the  analysis,  and  all  targets  were  considered. 
The  red  trace  shows  that  the  overall  GEM-3  performance  improved  very  slightly 
from  the  EMI-only  analysis  presented  in  Figure  19.  The  mag-assisted  ROC  curve 
performance  is  slightly  higher,  but  the  false  alarm  rates  at  the  operating  Pd/FAR 
are  still  high.  The  operating  Pd  did  increase  sufficiently  to  meet  Tier  II  require¬ 
ments  for  all  three  sites.  The  maximum  Pd,  however,  was  lowered  by  10  percent 
in  Area  2  and  5 -percent  in  Area  3. 

In  Figure  21,  the  blue  traces  show  that  overall  EMMS  Pd  detection  perform¬ 
ance  actually  decreased  with  the  addition  of  the  mag  data.  Comparison  of  these 
results  with  those  in  Figure  19  shows  that,  for  all  three  areas,  the  ROC  curve 
performance  is  lowered  when  the  mag  data  are  included  in  the  analysis.  In  addi¬ 
tion,  the  operating  Pd/FARs  are  significantly  lower  because  of  both  a  decrease  in 
Pd  and  an  increase  in  the  false  alarms,  and  the  operating  Pd/FARs  fail  to  meet 
Tier  II  requirements  in  all  three  areas.  The  maximum  Pd  is  also  slightly  lower  in 
all  three  areas. 

In  Figure  21,  the  yellow  traces  show  that  overall  EM-63  performance 
improved  slightly  over  the  results  presented  in  Figure  19.  The  ROC  curve  per¬ 
formance  improved  slightly  for  Areas  1  and  3  and  significantly  for  Area  2.  False 
alarms  for  Area  2  were  reduced  by  nearly  a  factor  of  2  but  only  slightly  reduced 
for  the  other  areas.  The  operating  Pd,  however,  is  lower  than  those  obtained  with¬ 
out  the  mag  data  and  failed  to  meet  Tier  II  requirements  for  all  three  areas.  The 
maximum  obtainable  Pd  was  lower  in  all  cases. 

Comparison  of  the  results  across  the  three  demonstrators  indicates  that  any 
enhancements  resulting  from  the  addition  of  mag  data  are  generally  minor,  and  in 
many  cases,  the  addition  of  mag  data  actually  degraded  system  performance. 
NAEVA  demonstrated  the  largest  performance  improvement  (Area  2)  over  the 
EMI-only  analysis  and  has  the  overall  best  ROC  curve  performance.  NRL  was 
slightly  worse  than  NAEVA  in  Areas  1  and  2,  and  significantly  worse  in  Area  3. 
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Figure  21 .  Detection  performance  of  GEM-3,  EMMS,  EM-63  system  (MAG 
w/20-mm  results) 
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The  GEM-3  ROC  curve  performance  improved  slightly  but  is  still  well  below 
that  of  the  other  two  demonstrators.  The  largest  impact  observed  was  on  the 
operating  Pd  point,  which  improved  for  the  GEM-3  while  decreasing  for  the 
other  two  demonstrators.  As  a  result,  the  GEM-3  was  the  only  system  meeting 
Tier  II  requirements.  There  appears  to  be  no  trend  or  reasonable  explanation  for 
the  widely  varying  effects  resulting  from  the  addition  of  mag  data  in  the  analysis. 
For  example,  it  would  be  expected  that  mag  data  would  significantly  enhance  the 
EMI  results,  especially  in  Area  3  where  the  geologic  magnetic  noise  is  minimal 
but  the  data  do  not  support  this  hypothesis. 

Figure  22  shows  the  detection  performance  of  the  three  systems  against 
57-mm  and  larger  targets  after  demonstrators  were  allowed  to  integrate  magne¬ 
tometer  data  into  their  analysis  of  the  EMI  data.  The  purpose  of  this  analysis  is  to 
quantify  performance  improvements  in  midsized  UXO  detection  from  magne¬ 
tometer  data  under  varying  clutter  conditions. 

In  Figure  22,  the  red  traces  show  that  the  overall  GEM-3  results  with  the  mag 
data  included  are  radically  different  from  the  EMI-only  analysis  presented  in 
Figure  20.  There  seems  to  very  little  correlation  between  the  mag-enhanced  and 
the  EMI-only  ROC  curves,  and  these  results  are  more  closely  correlated  with  the 
original  onsite  analysis  that  included  the  20-mm  projectile  targets.  It  appears  that 
the  significant  change  seen  in  Figure  21  was  because  of  a  decision  (or  threshold) 
that  resulted  in  the  elimination  of  a  large  number  of  anomalies  from  the  dig  lists 
and  that  this  decision  was  reversed  in  the  course  of  the  subsequent  mag-EMI 
analysis.  Comparison  of  the  ROC  curves  in  these  two  figures  shows  that  the  mag- 
EMI  analysis  includes  almost  twice  as  many  objects  as  the  EMI-only  analysis. 
The  initial  slopes  of  the  two  sets  of  curves  are  very  similar,  but  the  mag-EMI  set 
levels  off  and  continues  to  much  higher  false  alarm  counts.  The  operating  Pds  in 
Figure  22,  while  significantly  higher,  occur  at  much  higher  false  alarm  counts. 
The  mag-EMI  operating  Pds  exceed  the  Tier  II  requirements  for  all  three  Areas, 
whereas  all  of  the  Pds  from  the  EMI-only  data  failed  to  meet  them.  The  maxi¬ 
mum  Pds  are  also  considerably  higher  than  for  the  EMI-only  case  and  reach 
100-percent  for  all  three  areas. 

In  Figure  22,  the  blue  traces  show  that  the  overall  EMMS  performance 
improved  very  slightly  from  the  EMI-only  analysis  presented  in  Figure  20.  The 
ROC  curve  performance  for  all  three  areas  is  only  slightly  better  than  the  EMI- 
only  performance.  The  operating  Pd  performance  is  worse  for  the  mag-EMI  case 
since,  for  all  three  areas,  the  operating  Pds  are  slightly  lower  and  they  occur  at 
higher  false  alarm  counts.  Though  lower,  operating  Pds  still  meet/exceed  Tier  II 
requirements  in  all  three  areas.  The  maximum  achievable  Pd  is  slightly  lower  for 
Areas  1  and  3  and  unchanged  in  Area  2.  Overall,  EMMS  performance  was  not 
improved  by  incorporating  the  mag  data. 

In  Figure  22,  the  yellow  traces  show  that  the  overall  EM-63  performance  is 
almost  identical  to  the  EMI-only  analysis  presented  in  Figure  20.  Comparison  of 
the  two  sets  of  results  reveals  that  the  only  change  is  the  addition  of  a  few  false 
alarms  to  the  high  end  of  each  of  the  mag-EMI  ROC  curves.  As  a  result,  the  ROC 
curve  performance,  operating  Pds,  and  maximum  achievable  Pds  remain 
unchanged.  Overall,  the  EM-63  performance  was  not  significantly  improved  by 
the  addition  of  the  mag  data. 
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Figure  22.  Detection  performance  of  GEM-3,  EMMS,  EM-63  system  (MAG  w/o 
20-mm  results) 
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In  general,  the  addition  of  mag  data  had  very  little  effect  on  the  early  part  of 
the  ROC  curve  (i.e.  discrimination  ability).  The  effect  on  the  operating  Pds  was 
minor  for  NRL  an  NAEYA,  and  quite  significant  for  Geophex,  but  at  greatly 
increased  corresponding  false  alarm  rates.  The  primary  impact  on  the  NAEVA 
and  NRL  results  was  an  increase  in  the  number  of  anomalies  included  in  the  dig 
lists,  most  of  which  are  correctly  classified  as  clutter.  Comparison  of  this  set  of 
results  across  the  three  demonstrators  provides  very  limited  information  that  can 
support  conclusions  regarding  systems’  capabilities  or  to  arrive  at  meaningful 
conclusions  regarding  the  utility  of  multisensor  data.  In  addition,  the  results  are 
so  inconsistent,  that  they  will  not  support  the  objective  of  quantifying  EMI 
performance  improvements  due  to  the  addition  of  mag  data. 


Assessment  of  discrimination  and  identification  performance 

The  discrimination  and  identification  capabilities  of  UXO  systems  greatly 
affect  the  cost  and  residual  risks  associated  with  any  UXO  cleanup  operation. 

The  assessment  of  these  capabilities  for  the  three  advanced  systems  demonstrated 
at  JPG  is  included  in  this  section,  and  results  are  summarized  in  the  following 
figures  and  tables. 

The  discrimination  and  identification  performance  of  each  demonstrator  is 
based  on  the  UXO  vs  clutter  and  the  UXO  type  declarations  included  in  each  of 
the  required  prioritized  dig  lists.  The  results  presented  in  this  section  have  been 
adjusted  to  account  for  UXO-related  items  that  remained  in  Areas  1  and  2  from 
previous  JPG  demonstrations.  Figures  23  through  34  include  classification 
matrices  that  detail  and  summarize  the  detection/discrimination  performance  of 
each  demonstrator.  Each  classification  matrix  includes  the  following  entries: 

a.  The  entry  below  the  demonstrator’s  name  and  test  area  indicates  the  type 
of  analysis  used  to  obtain  the  results.  There  are  four  types:  Onsite  with 
20-mm  projectile  targets  included,  Offsite  without  20-mm  projectile 
targets  (EMI  data  only),  MAG  with  20-mm  targets  included  (includes 
mag  and  EMI  data),  and  MAG  without  20-mm  targets  (includes  mag  and 
EMI  data). 

b.  The  classes  across  the  top  of  the  matrix  are  the  actual  (ground-truthed) 
target  classes  of  the  items  emplaced.  The  projectiles  are  grouped  into 
three  classes.  The  small  projectile  class  contains  the  20-mm  projectile. 
The  medium  projectile  class  contains  the  57-,  76-,  and  the  105-mm  pro¬ 
jectiles.  The  large  projectile  class  contains  the  5-in.,  the  152-mm,  and  the 
155-mm  projectiles.  A  clutter  class  is  listed  as  a  separate  entry  and  corre¬ 
sponds  to  a  false  alarm  if  classified  as  a  UXO  target  and  to  a  correct  dis¬ 
crimination  when  classified  as  non-UXO  by  the  demonstrator. 

c.  The  classification  column  on  the  left  side  of  the  matrix  lists  the  demon¬ 
strator’s  declaration  for  each  detected  UXO  item.  Again,  the  projectiles 
are  grouped  into  three  classes  as  previously  described  in  the  target 
classes. 
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d.  The  classification  matrices  for  dig  sheets  that  exclude  the  20-mm  pro¬ 
jectiles  do  not  have  a  small  projectile  class  for  the  contractor  classifi¬ 
cation  or  for  the  target  classification.  Any  target  declaration  by  the 
demonstrators  that  corresponds  to  a  20-mm  projectile  was  included  as  a 
false  alarm  (i.e.,  considered  as  a  clutter  item). 

e.  The  totals  in  the  right  hand  column  of  the  matrix  correspond  to  the  total 
number  of  items  declared  by  the  contractor  as  a  particular  class. 

f.  The  “%  Classified”  row  indicates  the  percentage  of  detected  targets,  for  a 
given  target  class,  that  were  correctly  classified. 

g.  The  “%  Classified  by  Class”  indicates  the  percentage  of  detected  targets, 
for  the  given  classes  of  projectile,  mortar,  and  rocket,  that  were  correctly 
classified. 

h.  The  “%  of  Total  Detected  Targets  Classified  as  Nonordnance  Low/Med 
Confidence”  is  the  percentage  of  detected  ordnance  that  the  contractor 
incorrectly  classified  as  nonordnance  with  low  or  medium  confidence. 

i.  The  “%  of  Total  Detected  Targets  Classified  as  Nonordnance  High 
Confidence”  is  the  percentage  of  detected  ordnance  that  the  contractor 
incorrectly  classified  as  nonordnance  with  high  confidence.  This  classifi¬ 
cation  error  carried  the  highest  cost  penalty  (equivalent  to  the  cost  of  a 
complete  resurvey  of  the  area). 

Note  that  if  a  detected  target  was  declared  as  belonging  to  more  than  one 
class,  only  the  first  class  in  the  dig  list  has  been  included  the  classification 
matrix.  For  example,  NRL  generally  declared  multiple  classes  for  their  detections 
and  their  dig  list  always  started  with  a  projectile;  therefore,  no  classifications  for 
mortars  were  entered  into  these  classification  matrices. 

The  onsite  analysis  results  presented  in  Figure  23  indicate  that  overall,  the 
GEM-3  demonstrated  poor  capability  to  discriminate  ordnance  items  from  clutter 
across  all  target  types  and  at  all  three  areas.  A  total  of  17  ordnance  items  were 
declared  as  nonordnance  with  high  confidence,  which  in  an  actual  UXO  cleanup 
would  have  resulted  in  leaving  live  ordnance  in  the  ground. 

The  results  presented  in  Figure  24  indicate  that  the  EMMS  was  fairly  effec¬ 
tive  in  correctly  discriminating  ordnance  from  nonordnance  items,  but  still  incor¬ 
rectly  declared  a  total  of  seven  ordnance  items  (including  five  20-mm  projectiles) 
as  nonordnance  with  high  confidence.  Identification  performance  was  poor 
across  all  target  types. 

The  results  presented  in  Figure  25  show  that  the  NAEVA  EM-63  system 
demonstrated  the  highest  discrimination  performance  based  on  the  onsite  dig 
lists.  Only  two  items  were  classified  as  nonordnance  with  high  confidence  (none 
in  Area  1).  Identification  performance  was  poor  across  all  target  types. 

The  offsite  results  using  EMI  only  and  excluding  the  20-mm  projectiles  are 
presented  in  Figure  26  and  show  that,  while  the  number  of  false  alarms  were 
significantly  reduced,  the  discrimination  and  identification  performance  of  the 
GEM-3  remained  extremely  poor.  For  example,  out  of  a  total  number  of  16  items 
in  Area  1  that  were  declared  as  nonordnance  with  high  confidence,  8  items  were 
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Figure  26.  Classification  matrix  for  GEM-3  (EMI  only  w/o  20  mm) 
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actually  emplaced  ordnance  items.  A  total  of  13  such  items  were  incorrectly 
classified  over  the  three  test  areas.  With  the  possible  exception  of  the  4.2-in. 
mortars  (which  were  too  few  from  which  to  draw  valid  conclusions),  there  was 
no  identification  capability  demonstrated. 

The  offsite  discrimination  performance  of  the  EMMS,  which  is  summarized 
in  Figure  27,  showed  significant  improvement  from  the  onsite  results.  A  major 
reason  for  this  improvement  is  due  to  the  fact  that  5  of  the  7  UXO  items  errone¬ 
ously  classified  as  high  confidence  clutter  in  the  onsite  results  were  in  fact  20 
mm  ordnance  items.  Since  the  offsite  analysis  was  intended  to  ignore  20  mm 
sized  target,  the  majority  of  the  incorrectly  classified  items  were  easily  elimi¬ 
nated.  As  a  result,  only  one  item  (in  Area  2)  was  incorrectly  classified  as 
nonordnance  with  high  confidence.  There  was  no  demonstrated  identification 
capability. 

The  offsite  results  for  NAEVA  are  presented  in  Figure  28,  and  indicate 
significant  discrimination  capability.  Only  one  ordnance  item  (in  Area  1)  was 
misclassified  as  nonordnance  with  high  confidence.  There  was  no  demonstrated 
identification  capability. 

The  next  set  of  six  figures  summarizes  the  offsite  analysis  conducted  after 
release  of  the  MTADS  magnetometry  data.  In  general,  the  additional  information 
from  the  mag  data  did  not  improve  the  identification  capabilities  of  the  three 
systems. 

Comparison  of  the  discrimination  performance  across  the  three  systems 
shows  that  NAEVA  demonstrated  the  best  capability  to  reliably  discriminate 
ordnance  from  clutter.  NRL  demonstrated  considerably  lower  discrimination 
capability,  and  Geophex  Ltd.  demonstrated  very  poor  discrimination  capability. 
None  of  the  systems  could  be  considered  to  have  demonstrated  capability  to 
identify  ordnance  items  either  by  type  or  by  class. 


Summary  of  detection,  discrimination,  and  identification 
performance 

In  order  to  facilitate  the  comparison  of  detection,  discrimination,  and  identi¬ 
fication  performance  across  the  three  demonstrators,  as  well  as  with  the  “mag 
and  flag”  results,  where  appropriate,  the  dig  list  information  is  summarized  again 
in  Tables  2  through  5.  It  should  be  noted  that  in  these  tables,  unlike  the  previous 
classification  matrices,  the  dig  list  declarations  were  interpreted  to  accept  the 
correct  one  of  the  first  two  choices  listed.  For  example,  if  the  dig  list  specified 
57-mm  projectile/60-mm  mortar  for  an  actual  mortar  target,  credit  was  given  in 
the  mortar  class. 


Performance  of  demonstrators  against  overlapping  targets 

Each  demonstration  area  included  three  UXO  targets  that  had  clutter  items  in 
close  proximity  so  that  their  magnetic  and  EMI  signatures  would  overlap.  The 
purpose  of  these  closely  spaced  targets  was  to  evaluate  the  spatial  resolution  of 
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Figure  27.  Classification  matrix  for  EMMS  (EMI  only  w/o  20  mm) 
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Figure  28.  Classification  matrix  for  EM-63  (EMI  only  w/o  20  mm) 
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Figure  30.  Classification  matrix  for  EMMS  (MAG  w/20-mm  results) 
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Figure  31.  Classification  matrix  for  EM-63  (MAG  w/20-mm  results) 
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Figure  32.  Classification  matrix  for  GEM-3  (MAG  w/o  20-mm  results) 
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Figure  33.  Classification  matrix  for  EMMS  (MAG  w/o  20-mm  results) 
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2 

9 

14 

Non-ordnance  Low/Med 

1 

1 

1 

18 

21 

Non-ordnance  High 

1 

23 

24 

Total 

9 

7 

5 

5 

2 

4 

85 

117 

117 

%  Classified 

33.33% 

42.86% 

0.00% 

40.00% 

100.00% 

50.00% 

. 

. : . 1 

%  Classified  by  Class 

.. 

%  of  Total  Detected 

%  of  Total  Detected 

Projectile 

37.50% 

Targets  Classified  as 

Targets  Classified  as 
Non-ordnance  Hieh 

Mortar 

33.33% 

Non-ordnance  Low/Med 

. 

Rocket 

50.00% 

Confidence 

9.38% 

Confidence 

. w" . 

3.13% 

NAEVA  Area  2 

Classification  Matrix 

MAG  without  20  mm 

Target  Classification 

Projectile  Medium 

Projectile  Large 

Mortar 

Mortar 

Mortar 

Rocket 

. ; . i . 

Contractor  Class iflcation 

57  mm  thru  105  mm 

5  in  thru  155  mm 

60  mm 

81  mm 

4.2  in 

2.75  in 

Clutter 

Total 

Projectile  Medium 

57  mm  thru  105  mm 

1 

1 

j 

8 

11 

Projectile  Large 

5  in  thru  155  mm 

4 

1 

2 

10 

17 

Mortar  60  mm 

1 

2 

1 

9 

13 

Mortar  81  mm 

1 

2 

9 

12 

Mortar  4.2  in 

1 

6 

7 

Rocket  2.75  in 

1 

2 

3 

6 

Non-ordnance  Low/Med 

1 

1 

20 

22 

Non-ordnance  High 

28 

28 

Total 

4 

5 

5 

6 

1 

2 

93 

116 

116 

%  Classified 

25.00% 

80.00% 

40.00% 

33.33% 

100.00% 

100.00% 

%  Classified  by  Class 

J 

%  of  Total  Detected  i 

%  of  Total  Detected 

Projectile 

55.56% 

Targets  Classified  as 

Targets  Classified  as 

Mortar 

41.67% 

Non-ordnance  Low/Med 

Non-ordnance  Hiffh 

Rocket 

100.00% 

Confidence 

8.70%! 

Confidence 

0.00% 

. ? . * 

NAEVA  Area  3 

MAG  without  20  mm 

Classification  Matrix 

Target  Classification 

1 

.  ' . 

Projectile  Medium 

Projectile  Large 

Mortar 

Mortar 

Mortar 

Rocket 

Contractor  Classification 

57  mm  thru  105  mm 

5  in  thru  155  mm 

60  mm 

81  mm 

4.2  in 

2.75  in 

Clutter 

Total 

Projectile  Medium 

57  mm  thru  105  mm 

1 

1 

6 

8 

Projectile  Large 

5  in  thru  155  mm 

, 

2 

9 

12 

Mortar  60  mm 

1 

9 

10 

M  ortar  81  mm 

1 

2 

3 

8 

14 

Mortar  4.2  in 

3 

3 

Rocket  2.75  in 

1 

7 

8 

Non-ordnance  Low/Med 

1 

9 

10 

Non-ordnance  High 

13 

13 

Total 

3 

2 

4 

4 

1 

64 

78 

78 

%  Classified 

33.33% 

50.00% 

0.00% 

75.00% 

0.00% 

100.00% 

. 1 . . . ; 

. 

i 

%  Classified  by  Class 

%  of  Total  Detected 

%  of  Total  Detected 

Targets  Classified  as 

Non -ordnance  High 

Confidence  0.00% 

Projectile 

40.00% 

Targets  Classified  as 
Non-ordnance  Low/Med 
Confidence  7.14% 

. . .  d 

...........  .L  . 

Mortar 

37.50% 

Rocket 

100.00% 

Figure  34.  Classification  matrix  for  EM-63  (MAG  w/o  20-mm  results) 
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Table  2 

Demonstrators’  Ability  to  Detect  and  Discriminate  Targets  by  Class  -  Onsite  Analysis 
Results 


Total  Score 


Area  2 

Area  3 

Actual  -  Found  -  Match 

Actual  -  Found  -  Match 

27 
12 

04  03 


Note:  P  -  Projectile,  M  -  Mortar.  R  -  Rocket. 


16 

08 

- 

12 

11 

- 

02 

02 

- 

Table  3 

Demonstrators’  Ability  to  Detect  and  Discriminate  Targets  by  Class  -  Offsite  (excluding 
20  mm)  Analysis  Results 


Demonstrator  Class 

Area  1 

I  Area  2  I 

Area  3 

Actual  -  Found  -  Match 

Actual  -  Found  -  Match 

Geophex 


17 

16 

12 

12 

04 

04 

Total  Score 


Note:  P  -  Projectile,  M  -  Mortar,  R  -  Rocket. 


Table  4 

Demonstrators’  Ability  to  Detect  and  Discriminate  Targets  by  Class  -  Joint  Mag/Em  (with 
20  mm)  Analysis  Results 


Area  1 

Area  2 

Area  3 

Total  Score 

Actual  -  Found  -  Match 

Actual  -  Found  -  Match 

Actual  -  Found  -  Match 

Geophex 


Table  5 

Demonstrators’  Ability  to  Detect  and  Discriminate  Targets  by  Class  -  Joint  Mag/Em 
(excluding  20  mm)  Analysis  Results 


Area  1 

Area  2 

Area  3 

Total  Score  H 

Demonstrator 

Class 

Actual 

-  Found 

-Match 

Actual 

-  Found  - 

Match 

Actual 

-  Found  - 

Match 

Actual  -  Found 

-Match  I 

NRL 

P 

m 

04 

T^9 

EH 

E9 

mm 

M 

m 

Q 

05 

E9 

Eli 

mm 

R 

"9 

■H 

01 

01 

01 

07 

07 

01 

Geophex 

P 

17 

17 

05 

im 

04 

Hi 

E9 

33 

32 

13 

M 

12 

12 

09 

9 

eh 

08 

E9 

Eli 

33 

33 

23 

R 

04 

04 

0 

69 

EH 

0 

01 

01 

0 

07 

07 

0 

NAEVA 

P 

17 

16 

07 

E9 

EM 

06 

06 

05 

02 

33 

30 

15 

M 

12 

12 

08 

9 

07 

09 

08 

05 

33 

32 

R 

04 

04 

02 

69 

02 

01 

01 

01 

07 

07 

05  1 

|Note:  P  -  Projectile,  M  -  Mortar,  R  -  Rocket 


the  EMI  sensors  and  to  determine  the  robustness  of  the  various  discrimination, 
classification,  and  identification  techniques  employed. 

Figure  35  shows  the  abilities  of  the  demonstrators  against  large  UXO  targets 
with  overlapping  clutter  signatures.  In  all  cases,  the  demonstrators  reported  only 
one  target  which  indicates  limited  capability  of  resolving  close  targets  when  the 
signatures  are  fairly  large.  All  three  demonstrators  were  able  to  correctly  locate 
target  1-113  which  corresponds  to  a  105-mm  projectile,  and  two  (NRL  and 
NAEVA)  were  able  to  correctly  classify  the  target  as  a  projectile.  Geophex,  on 
the  other  hand,  incorrectly  declared  this  target  as  clutter  with  high  confidence. 

All  three  demonstrators  incorrectly  located  target  1-117,  a  152-mm  projectile,  at 
the  location  corresponding  to  the  clutter  item,  but  all  three  correctly  discrimi¬ 
nated  the  anomaly  as  a  UXO  target.  Only  NRL  correctly  classified  the  target  as  a 
projectile.  Target  1-121  was  a  155-mm  projectile  which  was  correctly  located, 
discriminated,  and  classified  as  a  projectile  by  both  NRL  and  NAEVA.  Geophex, 
on  the  other  hand,  declared  the  target  at  the  clutter  location  and  declared  it  as 
clutter  with  high  confidence.  NRL  correctly  identified  this  target  as  a  155-mm 
projectile. 

Figure  36  shows  the  capabilities  of  the  demonstrators  against  smaller  targets 
with  overlapping  clutter.  Target  2-131  consisted  of  an  81 -mm  mortar  which  was 
located  correctly  by  NRL  and  NAEVA,  and  correctly  discriminated,  classified 
and  identified  by  all  three  systems.  Target  2-120  consisted  of  a  60-mm  mortar, 
was  correctly  located  by  all  three  systems  and  discriminated  by  NRL  and 
NAEVA,  and  was  classified  as  a  mortar  only  by  NRL.  Target  2-166  consisted  of 
a  2.75-in.  rocket  warhead  and  was  the  only  target  in  this  area  that  was  resolved  as 
two  separate  objects  (by  NAEVA  and  Geophex).  Only  NAEVA  was  able  to 
discriminate  it  as  a  UXO  target,  and  NRL  and  Geophex  declared  it  as  a  clutter 
item  (with  medium  and  low  confidence,  respectively).  None  of  the  demonstrators 
were  able  to  correctly  classify  this  target  as  a  rocket. 

Figure  37  shows  the  capabilities  against  small  mortar  targets  with  over¬ 
lapping  clutter  signatures.  Target  3-68  consisted  of  a  60-mm  mortar  and  was 
correctly  located  and  resolved  into  two  separate  objects  only  by  Geophex.  Only 
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Groundtruth 

Geo 

phex 

NHL 

NA1 

EVA 

Area  1 

Found 

Declared 

Found 

Declared 

Found 

Declared 

2  -  Clutter 

113  -  Projectile 

113 

C-H 

113 

P 

113 

P 

18  -  Clutter 

1 17  -  Projectile 

18 

M 

18 

P 

18 

M 

184  -  Clutter 

121  -  Projectile 

184 

C-H 

121 

P 

121 

P 

Note:  P-Prpjectile.  M-Mortar,  R-Rocket,  C-L  -  Clutter  Low,  C-M  -  Clutter  Medium,  C-H  -  Clutter  Hiah 

Figure  35.  Overlapping  targets  in  Area  1 
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Groundtruth 

Geo 

phex 

NRL 

NAEVA 

Area  2 

Found 

Declared 

Found 

Declared 

Found 

Declared 

16  -  Clutter 

131  -  Mortar 

16 

M 

131 

M 

131 

M 

88  -  Clutter 

120  -  Mortar 

120 

C-L 

120 

M 

120 

P 

76  -  Clutter 

166  -  Rocket 

76/166 

C-L/P 

166 

C-M 

76/166 

P/P 

Note:  P-Projectile,  M-Mortar,  R-Rocket,  C-L  -  Clutter  Low,  C-M  -  Clutter  Medium,  C-H  -  Clutter  High 

Figure  36.  Overlapping  targets  in  Area  2 
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NA1 
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Area  3 

Found 

Declared 
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Found 

Declared 

6  -  Clutter 

68  -  Mortar 

6/68 

R/P 

6 

C-H 

6 

C-L 

50  -  Clutter 
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76 

R 

50 

C-M 

76 

M 

62  -  Clutter 

80  -  Rocket 

80 

C-H 

80 

M 

80 

M 

Note:  P-Projectile,  M-Mortar,  R-Rocket,  C-L  -  Clutter  Low,  C-M  -  Clutter  Medium,  C-H  -  Clutter  High 

Figure  37.  Overlapping  targets  in  Area  3 
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Geophex  was  able  to  discriminate  this  anomaly  as  a  UXO  but  incorrectly  classi¬ 
fied  it  as  a  projectile.  NRL  and  NAEVA  declared  this  object  as  clutter  with  high 
confidence  and  low  confidence,  respectively.  Target  3-76  also  corresponded  to  a 
60-mm  mortar  and  was  correctly  and  discriminated  by  Geophex  and  NAEVA.  In 
addition,  NAEVA  was  able  to  correctly  identify  the  signature  as  that  of  a  60-mm 
mortar.  NRL  incorrectly  declared  this  item  as  clutter  with  medium  confidence. 
Target  3-80  consited  of  an  81-mm  mortar  and  was  correctly  located  by  all  three 
demonstrators.  NAEVA  and  NRL  discriminated  the  anomaly  as  UXO,  and 
NAEVA  correctly  identified  the  target.  Geophex  incorrectly  declared  this  item  as 
clutter  with  high  confidence. 


Assessment  of  target  location  performance 

The  location  (x,y)  performance  of  each  of  the  three  demonstrators  was  evalu¬ 
ated  by  comparing  each  item  in  the  onsite  dig  list  with  the  ground  truth,  deter¬ 
mining  the  closest  item  (within  1  m)  to  an  emplaced  UXO  target  location,  and 
computing  the  error.  In  order  to  protect  the  ground  truth  information,  the  tables 
used  to  generate  this  evaluation  will  not  be  released  until  the  ESTCP  program 
office  decides  to  release  the  ground  truth.  The  ability  to  locate  clutter  items  was 
not  one  of  the  evaluation  criteria,  but  the  raw  data  was  available  for  such  analysis 
if  deemed  useful. 

The  evaluation  of  the  target  locations  reported  by  the  demonstrators  indicates 
that  the  GEM-3  achieved  the  highest  location  accuracy  for  all  the  targets  detected 
(all  three  test  areas).  The  average  horizontal  position  error  for  the  GEM-3  was 
16.6  cm.  The  EMMS  system  demonstrated  an  average  position  error  of  22.1  cm, 
and  the  NAEVA  EM-63  system  demonstrated  an  error  of  26.8  cm.  The  large 
majority  of  the  targets  detected  by  the  three  demonstrators  were  well  within  the 
0.5-m  error  radius  required  by  Kaho’olawe  cleanup  criteria. 

It  should  be  noted  that  two  UXO  targets  in  Area  1  (items  1-124  and  1-126) 
produced  the  largest  position  errors  (ranging  from  1 . 1  to  1.56  m)  for  all  demon¬ 
strators.  We  investigated  the  reason  for  these  errors  and  concluded  from  the  site 
preparation  and  previous  JPG  MTADS  mag  surveys  that  there  were  buried 
ferrous  objects,  outside  of  the  2-  by  2-m  areas  cleared  prior  to  target  emplace¬ 
ment,  that  were  large  enough  to  affect  the  signatures  of  the  emplaced  targets. 
Review  of  ground-truth  records  from  previous  JPG  demonstrations  indicate  that  a 
large  ferrous  object  had  been  placed  in  the  vicinity  of  each  of  these  locations. 

The  ability  of  the  demonstrators  to  estimate  the  depth  of  the  UXO  targets  is 
summarized  in  Table  6.  These  results  indicate  that,  while  the  performance  of 
each  demonstrator  varied  significantly  between  each  test  area,  the  mean  depth 
estimation  errors  were  well  within  the  0.5-m  allowable  error.  Overall,  the 
NAEVA  EM-63  system  achieved  the  best  depth  estimation  accuracy,  followed 
by  the  NRL  EMMS,  with  the  Geophex  GEM-3  demonstrating  the  largest  maxi¬ 
mum  and  mean  depth  estimation  errors. 
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Table 
UXO 1 

6 

forget  Depth  Estimat 

ion  Performance  of  the  C 

lemonstrators 

Area 

Demonstrator 

Minimum 
Error,  m 

Maximum 

Error,  m 

Mean 

Error,  m 

Std.  Dev 

1 

NAEVA  EM-63 

0.00 

0.63 

0.19 

0.14 

NRL  EMMS 

0.01 

0.65 

0.23 

0.17 

GEOPHEX  GEM-3 

0.01 

0.81 

0.20 

0.18 

2 

NAEVA  EM-63 

0.03 

0.72 

0.24 

0.19 

NRL  EMMS 

0.00 

0.86 

0.27 

0.24 

GEOPHEX  GEM-3 

0.04 

0.93 

0.30 

0.23 

3 

NAEVA  EM-63 

0.01 

0.76 

0.16 

0.21 

NRL  EMMS 

0.01 

0.37 

0.16 

0.10 

GEOPHEX  GEM-3 

0.02 

1.10 

0.31 

0.27 

Assessment  of  production  rate  performance 

As  part  of  the  demonstration  criteria,  the  Government  representatives  at  JPG 
recorded  man-hour  requirements  to  perform  all  onsite  tasks,  and  this  information 
is  available  from  the  ftp  site.  The  evaluation  factors  related  to  production  rate 
performance  are  described  in  the  demonstration  work  plan  and  listed  in  the 
Chapter  4  of  this  report.  Because  of  a  variety  of  problems  experienced  in  the  field 
by  some  of  the  demonstrators  (e.g.,  the  need  to  collect  a  new  target  signature 
library  onsite,  the  lack  of  capability  to  perform  onsite  analysis),  direct  compari¬ 
son  of  all  production  rate  factors  is  not  possible.  Since  survey  times  are  one  of 
the  more  important  production  factors,  and  accurate  information  from  all  the 
demonstrators  (including  “mag  and  flag”)  is  available,  it  has  been  decided  to  use 
this  factor  for  production  rate  performance  evaluation  and  comparison.  A  sum¬ 
mary  of  the  time  and  man-hours  required  to  survey  each  site  is  presented  in 
Table  7. 

From  the  data  summarized  in  this  table,  it  can  be  concluded  that  the  standard 
“mag  and  flag”  approach  achieved  an  average  production  rate  of  1  hectare  per 
5.97  hr  and  required  a  three-person  survey  crew.  The  best  performer  among  the 
advanced  technology  demonstrators  was  NRL  EMMS,  which  achieved  an 
average  of  1  hectare  per  6.56  hr,  and  required  a  field  survey  crew  ranging  from 
one  to  four  persons.  The  GEM-3  system  achieved  an  average  production  rate  of  1 
hectare  per  8.95  hr  and  required  a  two-person  survey  crew.  The  NAEVA  EM-63 
required  the  most  time,  with  an  average  of  13.34  hr  per  hectare  with  a  two-person 
crew. 


Technology  Comparison 

Overall,  discrimination,  classification,  and  identification  performance  of  all 
three  systems  was  lower  than  expected  and  significantly  lower  than  those  demon¬ 
strated  at  JPG  Phase  IV.  Some  obvious  reasons  for  the  decreased  performance 
include  the  facts  that,  unlike  at  JPG  IV,  the  demonstrators  did  not  have  prior 
access  to  the  clutter  items,  the  scenarios  were  more  realistic  and  representative  of 
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Table  7 

Production  and  Man-hours 

Demonstrator 

Area 

Survey  Work 

| 

GEOPHEX 

1 

Number  of  People 

1-2 

Time  on  Site,  hr,  min 

10:12 

Actual  Man-hours 

19:29 

2 

Number  of  People 

2 

Time  on  Site,  hr,  min 

9:08  I 

Actual  Man-hours 

18:16  3 

3 

Number  of  People 

2 

Time  on  Site,  hr,  min 

7:37 

Actual  Man-hours 

15:14 

NRL 

1 

Number  of  People 

1-4  1 

Time  on  Site,  hr,  min 

5:49 

Actual  Man-hours 

19:13  1 

2 

Number  of  People 

3-4 

Time  on  Site,  hr,  min 

7:55 

Actual  Man-hours 

25:20 

3 

Number  of  People 

3 

Time  on  Site,  hr,  min 

5:56 

Actual  Man-hours 

17:50 

NAEVA 

1 

Number  of  People 

2 

Time  on  Site,  hr,  min 

12:20 

Actual  Man-hours 

24:40 

2 

Number  of  People 

2 

Time  on  Site,  hr,  min 

14:19 

Actual  Man-hours 

28:38 

3 

Number  of  People 

2 

Time  on  Site,  hr,  min 

13:22 

Actual  Man-hours 

26:44 

EODT 

1 

Number  of  People 

3 

Time  on  Site,  hr,  min 

7:00 

Actual  Man-hours 

14:00 

2 

Number  of  People 

3 

Time  on  Site,  hr,  min 

6:10 

Actual  Man-hours 

12:20 

3 

Number  of  People 

3 

Time  on  Site,  hr,  min 

4:45 

Actual  Man-hours 

9:30 

actual  live  site  surveys,  and  the  systems  were  required  to  operate  in  a  wide  area 
search  mode  (rather  than  a  point  survey  mode  at  JPG  IV).  This  last  factor  signifi¬ 
cantly  reduced  the  number  of  data  samples  (which  resulted  in  decreased  spatial 
data  density  and  also  reduced  the  ability  to  perform  signal  averaging  to  increase 
signal-to-noise  ratios),  may  have  also  affected  the  available  signal  strength  levels, 
and  significantly  reduced  the  position  accuracy  from  that  achieved  during 
JPG  IV. 
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The  various  advanced  systems  demonstrated  varying  degrees  of  maturity. 

The  NRL  EMMS  demonstrated  the  highest  degree  of  maturity  and  preparation, 
and  conducted  the  field  surveys  and  onsite  analysis  with  no  problems.  In  contrast, 
the  GEM-3  and  EM-63  systems  demonstrated  a  lower  level  of  readiness  dining 
the  field  evaluations.  The  Geophex  team  experienced  a  sensor  failure  shortly 
after  starting  the  survey  and  was  forced  to  use  a  spare  GEM-3,  which  used  a 
different  coil  size.  This  required  the  collection  of  an  entirely  new  signature 
library  and  onsite  modifications  to  the  analysis  software.  The  processing  tech¬ 
niques  used  by  the  Geophex/AETC  team  were  not  fully  developed/tested  prior  to 
the  field  demonstrations  and  the  results  reflect  this  lack  of  preparation.  The 
NAEVA  team  arrived  onsite  without  the  capability  to  perform  the  analysis  tasks 
and  was  unable  to  demonstrate  onsite  processing.  The  processing  techniques  used 
by  the  NAEVA  team  were  also  not  fully  developed/tested  prior  to  the  field 
demonstrations. 
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Field  Cost  Performance 

Costs  associated  with  each  field  task  were  computed  by  applying  the  cost 
factors  described  in  the  Demonstration  Work  Plan  and  are  detailed  in  Table  8  and 
summarized  in  Table  9.  It  should  be  noted  that  the  NAEVA  cost  breakdown  does 
not  include  the  major  portion  of  the  data  analysis  since  it  was  conducted  offsite 
and  could  not  be  tracked.  The  only  analysis  costs  attributable  to  NAEVA  were 
those  required  to  download  data  to  laptops.  These  laptops  were  then  used  to 
transmit  these  data  to  the  workstations  offsite  for  analysis  and  development  of 
dig  lists  and  georeferenced  maps.  As  a  result.  Table  9  also  includes  a  tabulation 
of  the  total  field  costs  with  and  without  data  analysis  in  order  to  facilitate  the 
comparison  of  costs  across  systems.  It  should  also  be  noted  that  since  the  three 
demonstrated  systems  were  man-portable,  with  similar  support  equipment  and 
capital  cost  requirements,  it  was  assumed  that  mobilization/demobilization  and 
life-cycle  costs  would  be  equal  and  could  be  omitted  from  this  cost  performance 
evaluation. 

Analysis  of  these  tables  show  that  the  field  costs  of  the  three  demonstrators 
were  fairly  close,  with  NRL  demonstrating  the  lowest  costs,  followed  by 
Geophex  being  slightly  lower  than  NAEVA  when  analysis  costs  are  neglected. 
The  baseline  “mag  and  flag”  field  work  conducted  by  EODT  was  considerably 
lower  than  all  three  demonstrators,  but  it  should  be  noted  that  EODT  was  not 
required  (nor  capable)  of  providing  georeferenced  sensor  maps,  prioritized  dig 
lists,  and  target  discrimination/classification.  “Mag  and  flag”  also  failed  to  reach 
80-percent  Pd  at  any  of  the  three  test  areas,  had  significantly  higher  false  alarms 
at  the  high  magnetic  background  areas,  and  would  not  have  met  the  Kaho’olawe 
clearance  requirements. 


Weighted  Field  Cost  Performance 

Table  10  summarizes  the  operational  costs  of  the  demonstrator  systems  after 
the  cost  penalties  described  in  Chapter  4  were  applied.  These  penalties  consisted 
of  $200  for  each  false  alarm  (clutter  item  selected  for  digging  by  the  demon¬ 
strator),  and  the  cost  of  a  complete  resurvey  for  one  or  more  UXO  targets  missed 
or  erroneously  classified  as  clutter  with  high  confidence.  This  table  highlights  the 
fact  that  false  alarms  have  (by  a  large  margin)  the  greatest  impact  on  the  cost 
performance  of  each  system.  Table  10  indicates  that  all  three  demonstrators  were 
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Table  8 

Breakdown  of  Field  Costs 

Time 

Demonstrator 

Area 

Categories 

Cost 

(hrs,  min) 

Cost  to  Job 

Geophex 

1 

Supervisor 

$95.00 

25:50 

$2,454.16 

Data  Analysis 

$57.00 

30:42 

$1,749.90 

Logistic/Field  Setup 

$28.50 

16:38 

$474.05 

Logistic/Field  Survey 

$28.50 

19:29 

$555.27 

Logistic/Field  Downtime 

$28.50 

0:46 

$21,385 

Logistic/Field  Resurvey 

$28.50 

0:00 

$.00 

Rain  Delay 

$28.50 

0:00 

$.00 

Total 

$5,255.23 

2 

Supervisor 

$95.00 

27:48 

$2,641.00 

Data  Analysis 

$57.00 

15:35 

$888.25 

Logistic/Field  Setup 

$28.50 

25:34 

$728.65 

Logistic/Field  Survey 

$28.50 

18:16 

$520.60 

Logistic/Field  Downtime 

$28.50 

2:00 

$57.00  I 

Logistic/Field  Resurvey 

$28.50 

7:50 

$223.25 

Rain  Delay 

$28.50 

1:40 

$0.00 

Other  Calibration 

$28.50 

1:15 

$35.63 

Total 

$5,094.38 

3 

Supervisor 

$95.00 

15:56 

$1,513.66 

Data  Analysis 

$57.00 

14:54 

$849.30 

Logistic/Field  Setup 

$28.50 

11:00 

$313.50 

Logistic/Field  Survey 

$28.50 

15:14 

$434.15 

Logistic/Field  Downtime 

$28.50 

0:00 

$.00 

Logistic/Field  Resurvey 

$28.50 

0:00 

$.00 

Rain  Delay 

$28.50 

0:00 

$.00 

Total 

$3,111.09 

NRL 

1 

Supervisor 

$95.00 

13:56 

$1,323.66 

Data  Analysis 

$57.00 

12:05 

$688.75 

Logistic/Field  Setup 

$28.50 

6:58 

$198.55 

Logistic/Field  Survey 

$28.50 

24:28 

$697.30 

Logistic/Field  Resurvey 

$28.50 

5:09 

$146.78 

Total 

$3,055.04 

2 

Supervisor 

$95.00 

18:32 

$1,760.66 

Data  Analysis 

$57.00 

26:16 

$1,497.20 

Logistic/Field  Setup 

$28.50 

11:22 

$323.95 

Logistic/Field  Survey 

$28.50 

25:20 

$722.00 

Logistic/Field  Resurvey 

$28.50 

0:00 

$.00 

Total 

$4,303.81 

3 

Supervisor 

$95.00 

9:15 

$878.75 

Data  Analysis 

$57.00 

21:03 

$1,200.35 

Logistic/Field  Setup 

$28.50 

35:18 

$1,006.05 

Logistic/Field  Survey 

$28.50 

42:57 

$1,224.08 

Logistic/Field  Resurvey 

$28.50 

1:42 

$48.45 

Total 

$4,357.68 

(Continued)  || 
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Table  8  (Cc 

>nclud 

ed)  ”| 

Time 

Demonstrator 

Area 

Categories 

Cost 

hrs,  min 

Cost  to  Job 

NAEVA 

1 

Supervisor 

$95.00 

26:35 

$2,525.41 

Data  Analysis 

$57.00 

0:00 

$.00 

Logistic/Field  Setup 

$28.50 

19:12 

$547.20 

Logistic/Field  Survey 

$28.50 

24:40 

$703.00 

Logistic/Field  Resurvey 

$28.50 

6:44 

$191.90 

Rain  Delay 

$28.50 

0:00 

$.00 

Total 

$3,967.51 

2 

Supervisor 

$95.00 

11:45 

$1,116.25 

Data  Analysis 

$57.00 

0:50 

$47.50 

Logistic/Field  Setup 

$28.50 

2:30 

$71.25 

Logistic/Field  Survey 

$28.50 

28:38 

$816.05 

Logistic/Field  Resurvey 

$28.50 

6:53 

$196.17 

Rain  Delay 

$28.50 

0:00 

$.00 

Total 

$2,247.22 

3 

Supervisor 

$95.00 

33:01 

$3,136.59 

Data  Analysis 

$57.00 

1:10 

$66.50 

Logistic/Field  Setup 

$28.50 

16:06 

$458.85 

Logistic/Field  Survey 

$28.50 

26:44 

$761.90 

Logistic/Field  Resurvey 

$28.50 

9:06 

$259.35 

Rain  Delay 

$28.50 

1:30 

$0.00 

Total 

$4,683.19 

EODT 

1 

Supervisor 

$95.00 

6:30 

$617.50 

Data  Analysis 

$57.00 

0:00 

$.00 

Logistic/Field  Setup 

$28.50 

1:30 

$  42.76 

Logistic/Field  Survey 

$28.50 

10:30 

$  299.25 

Logistic/Field  Downtime 

$28.50 

0:00 

$.00 

Rain  Delay 

$28.50 

0:00 

$.00 

Total 

$  959.51 

2 

Supervisor 

$95.00 

6:00 

$  570.00 

Data  Analysis 

$57.00 

0:00 

$.00 

Logistic/Field  Setup 

$28.50 

0:30 

$  14.25 

Logistic/Field  Survey 

$28.50 

11:30 

$  327.75 

Logistic/Field  Downtime 

$28.50 

0:00 

$.00 

Rain  Delay 

$28.50 

0:00 

$.00 

Total 

$912.00 

3 

Supervisor 

$95.00 

5:15 

$  498.75 

Data  Analysis 

$57.00 

0:00 

$.00 

Logistic/Field  Setup 

$28.50 

1:30 

$  42.75 

Logistic/Field  Survey 

$28.50 

9:00 

$  256.50 

Logistic/Field  Downtime 

$28.50 

0:00 

$.00 

Rain  Delay 

$28.50 

0:00 

$.00 

Total 

$  798.00 

Table  9 

Total  Cost  for  All  Test  Areas 

Demonstrator 

Total  Cost  of  Field  Work 

Total  Cost  of  Field  Work 
Excluding  Data  Analysis 

NRL 

$  11,183.24 

$  7,854.44 

GEOPHEX 

13,507.27 

9,972.32 

NAEVA 

10,940.68 

10,783.93 

EODT 

2,669.51 

2,669.51 
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Table  1C 
Demons 
Leaving 

strator  Costs  Including  Penalties  for  False  Alarms  and 

UXO  Targets  in  Ground 

Area 

Number 

Demonstrator  1 

NRL 

Geophex 

NAEVA 

EODT 

1 

Cost  of  Survey 

$  3,055 

$  5,255 

$  3,968 

$  960 

Cost  of  Resurvey 

$  3,055 

$  5,255 

$  3,968 

$  960 

Cost  of  False  Alarms 

$14,200 

$18,200 

$15,000 

$26,900 

Total  Cost 

$20,310 

$28,710 

$22,936 

$28,820 

2 

Cost  of  Survey 

$  4,304 

$  5,094 

$  2,247 

$  912 

Cost  of  Resurvey 

$  4,304 

$  5,094 

$  2,247 

$  912 

Cost  of  False  Alarms 

$12,600 

$32,800 

$24,600 

$34,000 

Total  Cost 

$21,208 

$42,988 

$29,094 

$35,824 

3 

Cost  of  Survey 

$  4,358 

$  3,111 

$  4,683 

$  798 

Cost  of  Resurvey 

$  4,358 

$  3,111 

$  4,683 

$  798 

Cost  of  False  Alarms 

$13,000 

$31,000 

$15,800 

$20,600 

Total  Cost 

$21,716 

$37,222 

$25,166 

$22,196 

were  penalized  with  the  cost  of  a  resurvey  at  each  of  the  three  test  areas  because 
UXO  had  been  left  in  the  ground  as  a  result  of  miss-classified  or  missed  targets. 
NRL  demonstrated  significantly  lower  overall  costs  at  all  three  areas,  with 
NAEVA  slightly  lower  than  Geophex  (once  adjustments  for  lack  of  data  analysis 
costs  in  the  NAEVA  tabulations).  Comparison  with  the  baseline  “mag  and  flag” 
costs  indicates  that  the  best  performing  EMI  technologies  were  considerably 
more  cost  effective.  Even  though  the  EMI  systems  costs  include  cost  of  analysis 
in  both  the  survey  and  resurvey  cost  factors,  they  are  consistently  lower  than 
EODTs.  As  expected,  the  EMI  advantage  is  more  significant  in  Areas  1  and  2, 
which  have  significant  levels  of  magnetic  noise  from  geologic  sources. 
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7  Regulatory  Issues — 
Compliance  and 
Acceptance 


Members  of  the  regulatory  community  who  are  aware  of  these  technology 
demonstrations  are  listed  in  Appendix  A. 
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8  Technology/Transition 
Implementation 


The  next  step  for  these  technologies  is  to  transition  to  demonstrations  at 
prepared  test  sites  on  Kaho’olawe  Island,  which  is  located  in  the  Hawaiian 
Islands  chain.  Kaho’olawe  Island  is  rich  in  basalt,  which  results  in  a  high  mag¬ 
netic  background,  making  the  detection  of  unexploded  ordnance  difficult.  It  is 
expected  that  the  best  performing  technology  will  be  rapidly  transitioned  to 
active  cleanup  operations  at  live  sites  throughout  the  island.  The  demonstrations 
will  take  place  during  the  spring/summer  of  FY01.  Currently,  range  clearance  at 
Kaho’olawe  is  being  conducted  by  Parson/UXB,  who  is  the  primary  contractor 
for  Naval  Facilities  Engineering  Command  (NAVFAC).  Mr.  Jim  Putnam, 

(808)  474-0559,  extension  224,  is  the  project  manager  for  NAVFAC. 


The  planned  schedule  for  demonstrating  and  transitioning  these  advanced 
EMI  technologies  are  as  follows: 


FY01  Milestones 

Est.  Completion 

Complete  Demonstration  Plan 

06/30/01 

Award  contracts  to  Technology  Demonstration  (1) 

07/31/01 

Complete  prep  of  Kaho’olawe  QA/QC  sites  with  realistic  target/clutter  mix 

08/31/01 

Complete  fieldwork  for  demonstration 

10/31/01 

Deliver  georeferenced  raw  and  processed  data  to  ESTCP 

12/31/01 

Complete  Draft  Cost  and  Performance  Report 

01/31/02 

Complete  Draft  Technical  Report 

02/28/02 

Complete  Final  Report 

04/30/02 
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9  Lessons  Learned 


The  three  advanced  technology  demonstrators  were  able  to  conduct  area 
surveys  under  realistic  field  conditions  and  achieved  high  Pd  (over  90  percent)  in 
the  presence  of  high  magnetic  background  clutter.  These  Pd’s  were  consistently 
higher  than  those  achieved  by  the  standard  “mag  and  flag”  surveys.  Two  of  the 
three  advanced  systems  (NAEVA  and  NRL)  achieved  higher  Pd  with  much 
lower  false  alarm  rates.  All  three  of  the  advanced  systems  were  able  to  meet  or 
exceed  the  required  Pd  of  0.85  for  Kaho’olawe  cleanup,  although  not  necessarily 
at  the  operating  point  selected.  All  of  the  advanced  systems  had  lower  numbers 
of  false  alarms  at  comparable  Pd  to  “mag  and  flag”. 

None  of  the  demonstrated  EMI  systems  experienced  false  alarms  from 
geologic  sources  and  thus  show  excellent  potential  to  transition  to  sites  such  as 
Kaho’olawe  where  over  30  percent  of  the  false  alarms  are  caused  by  high 
magnetic  geology. 

A  significant  conclusion  resulting  from  the  evaluation  of  the  demonstration 
results  is  that  the  “pseudo  ROC  curves”  developed  under  this  effort  provide  a 
practical  method  for  comparing  detection  performance  within  and  across  systems 
without  artificially  constraining  the  data  analysis  (e.g.,  requiring  single  threshold 
parameter). 

The  advanced  systems  demonstrated  significantly  improved  ROC-based 
performance  when  the  target  list  did  not  include  the  small  20-mm  projectiles. 
Generally,  substantial  decreases  in  false  alarms  are  accompanied  by  only  small 
decreases  in  Pd.  With  the  exception  of  one  demonstrator  (Geophex  in  Area  3),  all 
of  the  advanced  systems  met  the  Kaho’olawe  Tier  2  Pd  requirement.  NRL  and 
NAEVA  demonstrated  ability  to  select  an  operating  point  beyond  which  there 
were  very  few  ordnance  items  (almost  exclusively  clutter). 

The  demonstrated  discrimination,  classification,  and  identification  perform¬ 
ance  of  the  three  systems  was  lower  than  expected  based  on  previous  results 
obtained  under  less  realistic  test  environments.  When  20-mm  projectiles  are 
included  in  the  target  set,  the  best  performer  labeled  3  percent  of  the  ordnance  as 
clutter  with  high  confidence.  When  20-mm  projectiles  are  excluded,  both  NRL 
and  NAEVA  were  able  to  improve  their  discrimination  performance  so  that  only 
1  percent  of  the  ordnance  was  mislabeled  as  clutter  with  high  confidence.  None 
of  the  advanced  systems  demonstrated  the  capability  to  identify  ordnance.  It  is 
recommended  that  the  ground  truth  be  made  available  to  the  three  demonstrators 
prior  to  proceeding  to  the  next  demonstration  phase  in  order  to  incorporate 
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lessons  learned  from  this  effort.  It  is  expected  that  self  assessments  using  the 
ground  truth  will  lead  all  three  demonstrators  to  be  more  conservative  in  their 
anomaly  picking  and  especially  in  their  clutter/UXO  declarations.  In  addition, 
comparison  of  this  demonstration  results  with  previous  JPG  IV  data  indicates  that 
the  discrimination  and  identification  algorithms  currently  used  are  very  depen¬ 
dent  on  a  very  high  level  of  position  accuracy.  The  large  uncertainties  in  the  GPS 
position  information  as  well  as  the  errors  introduced  by  the  sensors  moving  over 
rough  terrain  significantly  degraded  the  systems’  discrimination  capabilities. 

A  significant  and  unexpected  result  of  the  demonstration  was  the  fact  that  the 
re-analysis  of  the  EMI  data  incorporating  additional  information  from  the  NRL 
magnetometer  data  did  not  improve  the  EMI-only  results.  The  mag/EMI  results 
were  so  inconsistent  that  no  strong  conclusions  can  be  drawn  as  to  the  utility  of 
multisensor  data.  As  a  result,  ESTCP  plans  to  provide  the  mag  and  EMI  survey 
data  to  a  number  of  researchers  involved  in  the  development  of  UXO  detection 
algorithms  in  order  to  fully  evaluate  any  performance  gains  derived  from  the 
joint  analysis  of  mag  and  EMI  data. 

The  demonstration  results  show  that  the  costs  of  survey  and  analysis  were  a 
factor  of  4  to  5  higher  for  the  advanced  systems  compared  to  standard  “mag  and 
flag”.  However,  the  overall  costs,  including  the  costs  of  excavating  false  alarms 
are  higher  for  “mag  and  flag”  than  for  any  of  the  advanced  systems. 

One  of  the  objectives  of  this  demonstration  was  to  identify  scenarios  where 
one  EMI  technology  may  be  better  suited  than  the  others.  The  performance  data 
acquired  during  these  tests  do  not  support  strong  conclusions  regarding  the  rela¬ 
tive  merits  of  the  three  systems.  As  a  result,  all  three  systems  will  be  included  in 
the  second  phase  demonstrations  at  Kaho’olawe  during  September  through 
November  2001 .  It  is  expected  that  the  significantly  more  difficult  environment 
present  at  Kaho’olawe  will  highlight  differences  in  performance  between  each  of 
the  systems. 

A  number  of  issues  arose  during  the  planning  and  execution  of  this  demon¬ 
stration  project  that  had  not  been  foreseen  and  which  affected  the  data  quality 
and  results.  Most  important  of  these  was  the  fact  that  a  variety  of  contracting 
mechanisms  were  used  to  fund  the  demonstrators,  and  each  of  these  contracts  had 
inconsistent  requirements.  For  example,  NAEVA  was  funded  directly  from 
ESTCP  under  an  existing  contract,  which  had  different  requirements  and  time¬ 
lines  than  our  Demonstration  Plan.  As  a  result,  NAEVA  was  not  ready  (or 
required  by  contract)  to  perform  onsite  analysis  of  the  EM-63  data  nor  to 
reanalyze  the  data  incorporating  the  MTADS/MMS  magnetometer  data.  It  is 
recommended  that,  in  the  future,  demonstrations  involving  multiple  Government 
and  commercial  demonstrators  be  funded  under  identical  scopes  of  work,  or  be 
conducted  with  adherence  to  a  common  demonstration  work  plan. 
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Appendix  A 
Points  of  Contact 


The  NAVEODTECHDIV  POC  for  this  project  is: 

Mr.  Hien  Dinh 

NAVEODTECHDIV 

2008  Stump  Neck  Road 

Indian  Head,  MD  20640-5070 

Phone:  (301)  7446850  ext.  267 

FAX:  (301)744-6947 

E-mail:  dinh@eodpoe2.navsea.navy.mil 

The  ERDC  POC  is: 

Dr.  Ernesto  Cespedes 
USAEWES 
3909  Halls  Ferry  Road 
Vicksburg,  MS  39180-6199 
Phone:  (601)  634-2655 
FAX:  (601)634-2732 
E-mail:  cespede@wes.armv.mil 

The  AEC  POC  is: 

Mr.  George  Robitaille 
U.S.  Army  Environmental  Center 
ATTN:  SFIM-AEC-P2/ETD 
Bldg  E4430 

Aberdeen  Proving  Ground,  MD  21010-5401 
Phone:  (410)436-6850 
FAX:  (410)436-6836 

E-mail:  George.Robitaille@aec.apgea.army.mil 

JPG  site  POC  is: 

Mr.  Ken  Rnouf 

Jefferson  Proving  Ground,  IN 

Phone:  (812)273-6075 

U.  S.  Fish  and  Wildlife  Service  at  JPG  POC  is: 
Dr.  Joe  Robb 

Jefferson  Proving  Ground,  IN 
Phone:  (812)273-0783 
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Important  points  of  contact  in  the  regulatory  and  user  community  who  have 
knowledge  of  the  demonstration  include: 


Interstate  Technology  Regulatory  Cooperation  (ITRC) 

ITRC  UXO  WORKGROUP 

Workgroup  Co-Leaders 

Jim  Austreng  (Team  Co-Lead) 

California  EPA 

Department  of  Toxic  Substances  Control 
10151  Croydon  Way,  Suite  3 
Sacramento,  CA  95827-2106 
P:  916-255-3702 
iaustren@dtsc.ca.gov 

Jennifer  Roberts  (Team  Co-Lead) 

Alaska  Department  of  Environmental  Conservation 
555  Cordova  Street 
Anchorage,  AK  99501 
P:  907-269-7553 

Jennifer_Roberts@envircon.state.ak.us 

Workgroup  Members 

David  Asiello 
US  Department  of  Defense 
3400  Defense  Pentagon 
Washington,  DC  20301-3400 

P:  703-697-7363 . F:  703-695-4981 

asielldj@acq.osd.mil 

Tim  Bahr 

Florida  Department  of  Environmental  Protection 
2600  Blair  Stone  Road 
Tallahassee,  FL  32399-2400 

P:  850-921-9984 . F:  850-922-4939 

Tim.Bahr@dep.state.fl.us 

Geoff  Cullison 

Office  of  the  Chief  of  Naval  Operations 
Navy  Environmental  Restoration  Program 
2211  South  Clark  Street 
Arlington,  VA  22202-3735 

P:  703-602-5329 . F:  703-602-2676 

cullison.geoffrey@hq.navy.mil 
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Jeff  Edson 

Colorado  Department  of  Public  Health  &  Environment 
4300  Cherry  Creek  Drive,  South 
Denver,  CO  80246 

P:  303-692-3388 . F:  303-759-5355 

Jeff.edson@state.co.us 

Dwight  Hempel 
Department  of  Interior 
Bureau  of  Land  Management 
1849  C  Street,  NW,  MS  1000LS 
Washington,  DC  20240 

P:  202-452-7778 . F:  202-452-7708 

dwight_hempel@blm.gov 

Aimee  Houghton 

Center  for  Public  Environmental  Oversight 
122  C  Street  NW,  Suite  700 

Washington,  DC  20001-2109 

P:  202-662-1888 . F:  202-628-1825 

aimeeh@cpeo.org 

Dave  Larsen 

Utah  Department  of  Environmental  Quality 

288  N.  1460  West 

Salt  Lake  City,  UT841 16 

P:  801-538-6749 . F:  801-538-6715 

Dlarsen@dea.state.ut.us 

Mike  Liberati 

DuPont  Corporate  Remediation  Group 
2000  Cannonball  Road 
Pompton  Lakes,  NJ  07442 

P:  302-892-7421  michael.r.liberati@usa.dupont.com 
Chris  Maurer 

Washington  Department  of  Ecology 
P.O.  Box  47600 
Olympia,  WA  98504 
P:  360-407-7223 
Cmau46 1  @ecy.  wa.gov 

Marshall  Nay 
TRW 

6001  Indian  School  Road 
Albuquerque,  NM  87110 

P:  505-998-8359 . F:  505-998-8125 

marshall.nav@trw.com 
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Eric  Noack 

Nevada  Division  of  Environmental  Protection 

333  Nye  Lane,  ES-1 1 1 

Carson  City,  NV  89710 

P:  775-687-4670  x3032 
enoack@ndep.carson-city.nv.us 

Steve  Nussbaum 

Illinois  Environmental  Protection  Agency 

1021  N.  Grand  Ave.  East 

Springfield,  IL  62794-9276 

P:  217-782-9803 . 

epa4129@epa.state.il.us 

. F:  217-524-3291 

Atul  R.  Patel 

Joint  UXO  Coordination  Office 

Attn:  AMSEL-RD-UXO-CO 

10221  BurbeckRoad 

Ft.  Belvoir,  VA  22060-5806 

P:  703-704-2609 . 

apatel@nvl.army.mil 

. F:  703  704-2074 

Rodney  Sobin 

Virginia  Department  of  Environmental  Quality 

629  East  Main  Street 

Richmond,  VA  23233 

P:  804-698-4382 . 

rsobin@deq.state.va.us 

. F:  804-698-4264 

Jerry  Stamps 

Environmental  Engineer 

South  Carolina  Department  of  Health  and  Environmental  Conservation 

2600  Bull  Street 

Columbia,  SC  29201 

P:  803-896-4285 . F:  803-896-4002 

stampsjm@columb34.dhec.state.sc.us 

Philip  Stroud 

Alabama  Department  of  Environmental  Management 

P.O.  Box  301463 

Montgomery,  AL  36130 

P:  334-271-7750 . 

pns@adem.state.al.us 

. F:  334-279-3050 

Jeff  Swanson 

Colorado  Department  of  Health  and  the  Environment,  Hazardous  Materials 
Division 

4300  Cheny  Creek  Drive,  South 

Denver,  CO  80246-1530 

P:  303-692-3416 . F:  303-759-5355 

Jeffrey.swanson@state.co.us 
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Julie  Wanslow 

New  Mexico  Department  of  the  Environment 
Harold  S.  Runnels  Building 
1190  St.  Francis  Dr. 

Santa  FeNM  87505-4182 
P:  505-827-1536 

julie_wanslow@nmenv.state.nm.us 
Greg  Zalaskus 

New  Jersey  Department  of  Environmental  Protection 

P.O.  Box  028 

401  East  State  Street 

Trenton,  NJ  08625-0413 

P:  609-984-2065 . F:  609-633-1545 

gzalasku@dep.state.nj  .us 
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Appendix  B 

Data  Archiving  and 

Demonstration  Plan(s) 


The  raw  data  from  each  of  the  three  demonstrators,  including  calibration 
data  and  descriptions  are  available  via  anonymous  ftp  from 
ftp://ipgdemo:estcpfvOO@hawk.wes.armv.mil.  Also  available  via  this  ftp  server 
are  georeferenced  maps  of  the  surveyed  areas  and  the  dig  lists  submitted.  A 
directory  map  of  the  archived  data  is  as  follows: 
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Appendix  C 

Demonstrator’s  Data  Collection 
and  Analysis  Plan(s) 


Prior  to  arriving  at  the  JPG  site,  the  three  demonstrators  were  required  to 
submit  a  description  of  their  proposed  data  collection  and  analysis  plans.  The 
plans,  as  approved  by  the  ESTCP  Program  Office,  are  included  in  this  appendix. 

C.1  Geophex  Ltd. 

Advanced  UXO  Detection/Discrimination  Technology 

Demonstration 

U.S.  Army  Jefferson  proving  Ground,  Madison,  Indiana 

Geophex  /  AETC  Data  Analysis  Process 
(Addendum) 


Data  Processing  and  Ranking 

Our  approach  for  identifying  specific  UXO  types,  as  well  as  for  discriminating 
between  UXO  and  clutter,  is  to  compare  the  sensor  response  to  a  library  of  UXO 
signatures.  This  requires  a  preprocessing  step  that  fits  the  data  to  a  dipole  model, 
from  which  estimates  of  the  target  orientation  and  depth  can  be  derived.  Once 
this  is  done,  the  target  can  be  effectively  rotated  into  a  principal  axis  coordinate 
system  and  the  spectral  response  in  this  system  can  be  compared  to  the  library 
responses.  Another  way  to  state  this  is  that  the  eigenvalues  of  the  magnetic 
polarizability  tensor  that  models  the  target  are  derived  from  the  data.  These 
eigenvalues  are  orientation  invariant  and  can  be  used  as  intrinsic  target 
signatures.  The  required  computation  to  perform  this  preprocessing  step  is  easily 
and  rapidly  carried  out  on  a  PC. 

Declarations  of  UXO  versus  non-UXO  will  be  determined  based  similarities  of 
the  derived  eigenvalues  and  library  signatures.  Confidence  rankings  will  be  based 
upon  the  degree  of  fit.  We  also  examine  the  amplitude,  spatial  pattern,  and 
measured  frequency  response  for  sanity  checks. 
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Anticipated  Acquisition  Parameters 


GEM-3  data  will  be  acquired  using  a  line  spacing  of  0.5  m  and  a  sample  rate  of 
around  1 0  Hz  (dependent  on  number  of  frequencies  selected).  Anticipated 
tracking  accuracy  of  our  system  is  approximately  10  cm.  Raw  sensor  output 
(GPS  and  GEM-3)  is  recorded  in  comma-separated  ASCII  files. 
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MTADS  Survey  Demonstration  Plan 

Jefferson  Proving  Ground,  Madison,  IN 
27  August  -  6  September  2000 


The  Survey  Site 

Three  1 -hectare  areas  have  been  prepared  at  the  JPG  16-hectare  and  the 
U.S.  Army  engineer  Research  and  Development  Center  (ERDC),  Vicksburg,  MS, 
test  sites,  as  described  in  the  “Site  Preparation  Plan”  (Ref.  1).  The  coordinates  of 
the  comers  of  the  sites  are  given  in  Table  C2.1  and  the  coordinates  of  four 
reference  monuments  are  given  in  Table  C2.2.  These  sites  have  been  seeded  with 
a  variety  of  ordnance  and  OE  scrap  items  as  described  in  the  Technology 
Demonstration  Plan  (Ref  2). 


The  MTADS  Demonstration  Schedule 

The  primary  goals  of  the  demonstration  are  to: 

•  Conduct  an  EM  survey  of  the  three  designated  sites. 

•  Analyze  the  EM  data  to  identify  and  locate  the  buried  targets. 

•  Prepare  dig  images  and  dig  lists  of  targets. 

•  Rank  all  analyzed  targets  with  probabilities  of  their  being  OE  scrap  or 
intact  ordnance  items  from  the  ordnance  list  in  Ref.  1. 

The  MTADS  EM  Man-Portable  adjunct  will  be  used  to  conduct  a  survey  of 
each  site  using  the  integrated  GPS  navigation  system.  The  MTADS  vehicular  EM 
array  will  be  available  onsite  as  a  backup  should  the  Man-Portable  system  not  be 
able  to  complete  the  survey. 

Following  completion  of  the  EM  survey,  an  MTADS  vehicular  survey  of  each 
site  will  be  conducted  using  the  magnetometer  array.  Magnetic  anomaly  images 
(similar  to  Figure  2,  Ref.  1)  will  be  prepared  for  each  site  and  provided  to  the 
sponsor  electronically  and  as  hard  copy. 

If  time  permits,  selected  data  will  be  taken  using  the  MTADS  Man-Portable 
Magnetometer  System.  These  data  will  not  be  used  specifically  for  this 
demonstration  but  will  be  used  for  comparative  purposes  in  evaluation  and 
reporting  on  the  performance  of  the  Man-Portable  Instrument. 


Hardware  Description 

We  will  deploy  the  Man-Portable  EM  system  with  the  0.5-  %  1.0-m  coil 
mounted  with  the  1  -m  dimension  cross-track.  This  system  is  now  a  standard  (off- 
the-shelf)  Geonics  instrument.  We  will  not  cite  all  the  operational  characteristics 
of  the  instrument.  As  we  will  be  using  it,  our  data-sampling  rate  will  be  10  Hz. 
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The  system  time  gate  (measured  by  ourselves)  is  280-470  nsec  following  shutoff 
of  the  transmit  pulse.  The  Trimble  GPS  navigation  system  provides  position 
updates  at  5  Hz.  The  horizontal  positional  accuracy  (location  of  the  GPS  mobile 
antenna)  is  about  5  cm.  An  attitude  sensor  has  been  added  to  the  EM  Man- 
Portable  survey  system  to  provide  pitch,  roll,  and  yaw  corrections  to  the  sensor 
coil  locations.  We  believe  that  the  predicted  horizontal  location  uncertainty  for 
good  signal-to-noise  targets  will  be  <30  cm  for  the  EM  Man-Portable  system  and 
<20  cm  for  the  magnetometer  Man-Portable  system. 

Data  processing  and  data  analysis  will  be  accomplished  using  two  PCs 
operating  the  MTADS  Data  Analysis  System  (DAS),  as  modified  for  this 
demonstration.  Additionally,  a  third  PC  will  be  on  site  and  used  to  create 
spreadsheets  and  Graphics  Products  that  are  unique  to  this  demonstration. 


Survey  Setups 

On  each  1 -hectare  site,  1.5-m  lanes  will  be  established,  oriented  in  the 
direction  best  suited  for  man-portable  survey.  Wooden  stakes  (1.5-m  spacing) 
will  be  driven  at  the  perimeter  and  strings  will  be  pulled  across  the  site  to 
establish  survey  lanes.  Survey  lane  spacing  of  0.5  m  will  be  used  with  the  Man- 
Portable  EM  system.  Vehicular  surveys  (if  they  are  done  with  either  the 
magnetometer  or  EM  arrays)  will  be  conducted  with  1 .75-m  lane  spacing.  Survey 
priorities  are  as  follows: 

•  All  EM  surveys  will  be  completed. 

•  The  vehicular  MTADS  survey  will  be  conducted  for  all  sites. 

•  Selected  Man-Portable  magnetometer  survey  data  will  be  taken. 


Target  Analysis  Approach 

EM  survey  data  will  be  analyzed  using  the  3-P  MTADS  DAS  fitting  routine. 
The  DAS  has  been  modified  to  allow  the  target  analysis  to  be  run  on  a  single  data 
set.  Additionally,  the  DAS  has  been  modified  to  also  run  the  point-dipole  EM 
fitting  routine  to  generate  a  target  caliber.  Selected  targets  have  been  pre¬ 
surveyed  and  analyzed  using  the  3-P  routine  and  information  has  been  used  to 
create  probability  functions  based  upon  comparison  to  the  three-dimensional 
ellipsoids  generated  from  individual  ordnance  library  data.  Additionally,  both 
magnetometry  and  EM  libraries  exist  using  vehicular  survey  data  and  traditional 
MTADS  baseline  fits  for  selected  ordnance  at  a  variety  of  depths  and  orientations. 
This  graphical  and  numerical  fitting  data  will  be  used  to  supplement  the  results 
and  predictions  of  the  3-p  analyses.  Additional  information  is  sometimes 
available  from  visual  cues  in  the  graphical  presentations  of  the  target  data  In 
magnetometry  data,  these  cues  are  used  routinely  to  discriminate  between  single¬ 
object  targets  and  clusters  of  smaller  items.  Magnetometry  target  data  are 
routinely  edited  to  remove  spurious  information  and  clutter  objects,  allowing  refit 
of  the  data.  This  approach  is  much  less  useful  with  EM  data.  All  this  analysis 
information  will  be  used  to  rank  order  the  target  list.  We  anticipate  that  initial 
assignments  and  ordering  will  be  made  based  upon  the  3-P  fits.  These  will  be 
refined  using  information  as  described  above. 
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Quality  Assurance  Plan 

NRL  has  prepared  a  Demonstration  Test  Plan  for  the  use  of  the  onsite 
personnel  as  we  always  do  on  any  MTADS  survey.  This  test  plan  contains 
operational  routines  and  check  lists  that  are  followed  to  assure  the  fidelity  of  the 
data.  This  approach  is  simpler  than  typical  of  MTADS  surveys  because: 

•  The  primary  survey  product  employs  only  a  single  sensor  (no  reference 
sensor  data  are  required  and  change-over  between  survey  modes  is  not 
anticipated  to  be  necessary  to  produce  the  primary  survey  products). 

•  No  onsite  target  prove-out  surveys  must  be  conducted  or  analyzed. 

Target  calibration  measurements  have  been  made  separately  before  the 
demonstration. 

•  To  assure  the  quality  of  the  data,  a  preliminary  short  stationary  data  set  is 
taken  at  the  beginning  of  each  day  using  a  calibrated  test  object.  These 
data  are  preprocessed  and  visualized  before  actual  survey  data  collection 
begins. 

•  Survey  data  are  downloaded  hourly  and  immediately  preprocessed  and 
visualized  to  assure  fidelity. 

•  Alterations  have  been  made  in  the  Man-Portable  data  collection  device  to 
allow  sensor  data  streams  to  be  visualized  in  real  time  in  the  field,  and 
audio  cues  have  been  installed  to  alert  the  surveyor  to  the  quality  of  the 
GPS  fix  being  provided  by  the  navigation  system. 

•  At  the  end  of  each  day.  GPS  data  downloaded  from  the  satellites  are  used 
to  predict  the  navigation  fix  qualities  as  a  function  of  time  for  the 
following  day.  Rest  breaks  and  meal  breaks  can  be  planned  around  poor 
GPS  periods.  If  surveys  adjacent  to  tree  lines  are  required,  various  GPS 
sky  masks  can  be  used  to  predict  the  best  times  for  survey  for  a  particular 
sky  view. 


MTADS  Support  Personnel 


Name 

Affiliation 

Job  Function 

J.R.  McDonald 

NRL 

PM,  Data  Analyst,  Field  Hand 

Herb  Nelson 

NRL 

Asst.  PM,  Field  Hand,  Data 
Analyst 

Bernard  Puc 

AETC 

Data  Analyst 

Larry  Koppe 

Geocenters 

Field  Hand 

Local  Laborer 

Nova 

Field  Hand 
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Table  C2.1 

UTM  Coordinates  of  JPG  Site  Corners 


Site  1 

Northing 

Easting 

Elevation 

NE  Comer 

4309710.266 

641434.444 

270.508 

SE  Comer 

4309610.653 

641443.029 

270.290 

SW  Comer 

4309602.063 

641343.410 

269.219 

NW  Comer 

4309701.673 

641334.818 

267.454 

Site  2 

NE  Comer 

4309750.232 

641713.068 

273.705 

SE  Comer 

4309650.606 

641721.640 

273.958 

SW  Comer 

4309642.015 

641621.998 

273.261 

NW  Comer 

4309741.643 

641613.412 

273.270 

Site  3 

NE  Comer 

4309791.612 

641593.714 

272.661 

SE  Comer 

4309891.174 

641585.150 

272.914 

SW  Comer 

4309899.745 

641684.758 

272.866 

NW  Comer 

4309800.142 

641693.340 

274.214 

Table  C2.2 


UTM  Coordinates  of  Rel 

erence  Monuments 

Reference  Monuments: 

Northing 

Easting 

Elevation 

(WES-0) 

4309790.274 

641578.800 

272.345 

Monument  1 

4309653.858 

641662.717 

273.853 

Monument  2 

4309425.242 

641545.826 

272.128 

Monument  3 

4309594.885 

641309.431 

272.128 
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C.3  NAEVA 


Proposed  Work  Plan,  Sept.  10  -  18, 2000 

EM63  Ordnance  Discrimination  Demonstration,  Jefferson 
Proving  Ground,  Ind. 

NAEVA  Geophysics,  ESTCP  Contract  DACA72-00-C-0009 

Introduction.  This  plan  describes  geophysical  investigations  proposed  for 
NAEVA  Geophysics’  EM63  Demonstration  (task  three)  at  the  Jefferson  Proving 
Ground,  Madison,  Indiana.  We  plan  to  accomplish  this  work  during  the  period 
September  10  through  19, 2000.  We  estimate  2  to  3  days  to  survey  each  of  the  1- 
hectare  seeded  areas  with  closely  spaced  (0.5  m)  traverse  lines  (8  days  total),  and 
one  additional  day  for  additional  bench  tests,  or  in  case  of  instrument  problems  or 
weather  delays. 

NAEVA  investigated  the  ordnance  discrimination  potential  of  the  Geonics 
EM61-3D  and  Protem  prototype  multi-channel  EM  equipment  at  JPG4.  The  new 
Geonics  EM63  instrument  has  been  modified  significantly  since  NAEVA’s 
initial  tests  in  October  1999,  when  it  first  became  available.  Measurements  of  the 
multi-channel  response  of  a  partial  suite  of  JPG  ordnance  at  all  orientations  in  air 
(beneath  the  EM63  instrument  on  a  test  bench)  were  made  during  NAEVA’s 
Blossom  Point  exercise  in  late  May  and  early  June  2000.  Further  tests  on 
additional  JPG  ordnance  items  will  be  done  during  the  week  following  Labor 
Day,  before  the  JPG  exercise.  No  unknown  target  can  be  identified  or  discrimi¬ 
nated  except  by  comparison  with  the  measured  response  of  an  exactly  equivalent 
item  (the  identical  model,  type,  and/or  construction).  The  technique  works  like 
‘fingerprints’,  in  that  each  specific  ordnance  type  has  a  distinct  multi-channel 
response  (at  least  for  each  10-degree  increment  of  orientation  between  vertical 
nose  up  and  nose  down),  which  must  be  in  the  discrimination  database.  The  suite 
of  sample  ordnance  items  provided  by  NAVEODTD  is  still  incomplete  (there  is 
no  3  inch  projectile  or  2.25  inch  rocket,  a  9”  60mm  sample  is  badly  split  by  a 
demo  charge,  which  makes  it  non-representative  of  unexploded  ordnance,  and  the 
105  mm  sample  is  an  unusual  ‘beehive’  or  flychette  type  which  is  not 
representative  of  any  105  mm  HE  or  armor  piercing  projectile). 

1)  Safety.  There  are  no  serious  safety  issues  at  the  JPG  test  site,  as  no  live 
ordnance  is  likely  to  be  present  on  the  surface  (after  surface  clearance  and  several 
years  of  intensive  geophysical  activity)  and  the  site  is  clear  of  serious  surface 
hazards.  NAEVA’s  field  data  acquisition  team  is  experienced  and  OSHA  safety 
trained,  all  normal  safety  procedures  and  precautions  will  be  observed  at  all 
times. 

2)  Personnel.  NAEVA’s  two-person  data  acquisition  team  (Alan  Mazurowski  and 
Todd  Nash)is  fully  experienced  with  the  EM63  and  GPS  equipment,  and  will 
exercise  with  the  modified  EM63  instrument  for  a  week  at  NAEVA’s  test  site 
before  the  Jefferson  Proving  Ground  demonstration.  The  demonstration  will  be 
supervised  in  the  field  by  Dr.  G.  Hunter  Ware  (geophysicist,  principal 
investigator).  Either  Dr.  William  Tompkins  (physicist)  or  Hunter  A.  Ware 
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(computer  scientist)  will  also  attend  at  least  a  part  of  the  exercise,  to  assist  with 
data  post-processing  and  to  verify  data  quality. 

3)  Test  Site.  The  Jefferson  Proving  Ground  site  is  familiar  to  NAEVA  from 
previous  JPG  demonstrations,  and  is  described  in  the  May,  2000,  Workplan 
distributed  by  Dr.  Ernesto  Cespedes.  The  coordinates  of  die  three  1 -hectare  test 
sites  were  described  in  Cespedes  E-mail  of  August  9, 2000. 

4)  Geophysical  Investigation  Methods.  The  test  site  will  be  investigated  by 
traversing  the  EM63  instrument  over  closely  spaced  lines  (approximately  north- 
south  and  spaced  approximately  0.5-m  apart),  traversing  at  slow  speeds,  so  as  to 
acquire  adequate  station  densities.  Extra  calibration  items  (iron  spheres)  will  be 
placed  on  the  ground,  probably  just  north  of  each  test  area,  for  amplitude 
calibrations  (top  and  bottom  coils). 

5)  Navigation.  Accurately  located  (RTK  differential  GPS)  locations  will  also  be 
acquired,  using  GPS  integration  software  developed  during  the  Blossom  Point 
exercise.  Tapes  or  ropes  will  be  placed  on  the  ground,  to  assure  an  even  and 
closely  spaced  distribution  of  traverse  lines. 

6)  Instrument  Calibration.  The  EM63  instrument  will  be  zeroed  (all  25  gates)  in 
the  air  or  over  non-responsive  soil  periodically.  The  response  to  standard 
calibration  objects  (generally,  3.5-in.  iron  spheres)  will  be  measured  in  each  data 
file  if  possible. 

7)  Data  Processing  (Target  Detection  and  Discrimination).  EMFIT  development 
software  will  be  used  for  data  post-processing  and  interpretation  (detection, 
target  location,  and  discrimination).  Geonics  DAT63  software  may  also  be  used 
for  some  purposes.  The  data  will  be  leveled,  de-spiked,  and  merged  with  GPS 
positions.  A  report  detailing  data  editing  and  digital  data  format  will  be  provided 
with  the  field  data.  Targets  will  be  detected  by  thresholding  (threshold  to  be 
determined  by  examination  of  local  terrain  noise).  A  prioritized  target  list  will  be 
developed,  based  upon  chi-squared  fit  of  bench  test  decay  curves  to  target  decay 
curves.  No  discrimination  is  possible  for  emplaced  (target)  items  for  which  we  do 
not  have  good  representative  bench  test  specimens.  Ordnance  items  may 
generaliy  be  detected  down  to  a  depth  of  10-x  diameter,  and  discriminated  to  a 
depth  of  about  5-x  diameter.  Target  objects  in  between  (detectable,  but  not 
discriminatable)  may  be  (a)  guessed  (randomly),  (b)  arbitrarily  declared 
ordnance,  (c)  arbitrarily  declared  nonordnance,  or  (d)  placed  in  a  separate 
‘unknown’  category.  If  you  guess,  you  get  half  of  the  targets  right!  These 
remarks  apply  to  time  or  frequency  domain  analysis. 


We  continue  to  make  concerted  efforts  toward  onsite  data  processing,  but  at  this 
time,  we  are  still  not  able  to  generate  a  prioritized  target  list  on  site,  using 
portable,  onsite  (laptop)  computers.  This  is  because  the  development  software 
routines  will  only  run  (or  only  run  fast  enough  for  large  data  sets),  on  our  larger 
office  computers.  We  expect  that  the  final,  compiled  software  will  run  onsite  (for 
small  data  sets),  but  our  business  model  continues  to  assume  that  data  will  be 
sent  over  the  internet  (by  ourselves  or  our  clients)  to  GPA’s  server  computer  for 
remote  processing,  as  this  will  be  faster  and  less  expensive  (services  always 
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available,  no  field  and  travel  expenses  for  data  processing).  We  do  plan  to 
submit  an  initial  prioritized  target  list  before  leaving  JPG.  We  will  be  happy 
to  provide  time  spent  on  this  remote  processing.  Additional  prioritized  target  lists 
((a)  for  other  target  suites  (for  example  without  20  mm)  and/or  (b)  prioritized  by 
decay  curve  shape  plus  principal  response  moments)  will  require  post-processing 
time  in  Charlottesville. 

8)  Quality  Control.  In  addition  to  the  zero  and  amplitude  calibration  procedures 
described  above,  principal  traverse  lines  over  target  and  calibration  items  will  be 
surveyed  in  both  directions  in  order  to  assure  that  the  data  are  repeatable  (and  lag 
corrected).  It  is  only  by  repeating  lines  in  opposite  directions  in  each  data  file  that 
proper  calibration  and  lag  (latency)  correction  can  be  verified.  We  strive  for  +/-  2 
mv  or  +/- 10%  peak  amplitude  and  +/-  2  cm  repeatability,  if  possible.  However,  it 
must  be  recognized  that  the  EM63  instrument  is  new,  and  its  instrument  noise 
levels  over  the  range  of  time  gates  is  not  yet  known. 

9)  Geophysical  Data.  All  raw  digital  data  files  will  be  leveled,  edited  (fiducialed, 
if  necessary),  and  converted  to  ASCII  x-y-zl  -z2-. . .  format  for  further  post¬ 
processing,  archiving,  and  for  presentation  to  ESTCP. 

10)  Performance  Goals.  We  hope  to  acquire  sufficient  bench  test  data  to  permit 
characterization  (discrimination)  of  all  emplaced  items  (inert  ordnance)  for  all 
inclinations.  Detection  depths  and  discrimination  depths  will  be  limited,  as  usual, 
by  local  terrain  noise.  We  have  not  yet  sampled  this  local  noise  (with  an  EM63), 
but  we  anticipate  detection  to  approximately  10  times  ordnance  diameter,  and 
discrimination  to  perhaps  5  times  ordnance  diameter  (discrimination  depth  is 
always  less  than  detection  depth).  All  data  will  be  repeatable  (to  instrument  and 
survey  noise  levels),  and  there  should  be  no  so-called  ‘false  positive’  anomalies 
tat  are  not  related  to  metal  objects  or  magnetic  rocks  or  soil. 


G.  Hunter  Ware,  Geophysicist 
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